A Tale of Two Flows: Cooperative Learning of Langevin Flow and Normalizing Flow Toward Energy-Based Model
Authors: Jianwen Xie
Yaxuan Zhu
Jun Li
Ping Li
Venue: ICLR 2022 Poster (2022)
Keywords: Langevin dynamics
energy-based model
normalizing flow
cooperative learning
short-run MCMC

Abstract: This paper studies the cooperative learning of two generative flow models, in which the two models are iteratively updated based on the jointly synthesized examples. The first flow model is a normalizing flow that transforms an initial simple density to a target density by applying a sequence of invertible transformations. The second flow model is a Langevin flow that runs finite steps of gradient-based MCMC toward an energy-based model. We start from proposing a generative framework that trains an energy-based model with a normalizing flow as an amortized sampler to initialize the MCMC chains of the energy-based model. In each learning iteration, we generate synthesized examples by using a normalizing flow initialization followed by a short-run Langevin flow revision toward the current energy-based model. Then we treat the synthesized examples as fair samples from the energy-based model and update the model parameters with the maximum likelihood learning gradient, while the normalizing flow directly learns from the synthesized examples by maximizing the tractable likelihood. Under the short-run non-mixing MCMC scenario, the estimation of the energy-based model  is shown to follow the perturbation of maximum likelihood, and the short-run Langevin flow and the normalizing flow form a two-flow generator that we call CoopFlow. We provide an  understating of the CoopFlow algorithm by information geometry and show that it is a valid generator as it converges to a moment matching estimator. We demonstrate that the trained CoopFlow is capable of synthesizing realistic images, reconstructing images, and interpolating between images.

Published as a conference paper at ICLR 2022
A TALE OF TWO FLOWS: COOPERATIVE LEARNING OF LANGEVIN FLOW AND NORMALIZING FLOW TOWARD ENERGY-BASED MODEL
Jianwen Xie, Yaxuan Zhu, Jun Li, Ping
Cognitive Computing Lab, Baidu Research 10900 NE 8th St. Bellevue, WA 98004, USA
kenny, yaxuanzhu.all a alvin, junli.sh04, pingli 1e ccom
ABSTRACT
This paper studies the cooperative learning of two generative flow models, in which the two models are iteratively updated based on the jointly synthesized examples. The first flow model is normalizing flow that transforms simple density to a target density by applying sequence of invertible transforma- tions. The second flow model is Langevin flow that runs finite steps of gradient- based MCMC toward an energy-based model. We start from proposing a gener- ative framework that trains an energy-based model with normalizing flow an amortized sampler to initialize the MCMC chains of the energy-based model. In each learning iteration, we generate synthesized examples by using a normalizing flow initialization followed by short-run Langevin flow revision toward the cur- rent energy-based model. Then we treat the synthesized examples as fair samples from the amergy-based model and update the model parameters with the maximum likelihood learning gradient, while the normalizing flow directly learns from the synthesized examples by maximizing the tractable likelihood. Under the short-run non-mixing MCMC scenario, the estimation of the energy-based model shown follow the perturbation of maximum likelihood, and the short-run Langevin flow and the normalizing flow form a two-flow generator that we call CoopFlow. We provide an understating of the CoopFlow algorithm by information geometry and show that is a valid generator as it converges to a moment matching estima- tor. We demonstrate that the trained CoopFlow is capable of synthesizing realistic images, reconstructing images, and interpolating between images.
initial
INTRODUCTION AND MOTIVATION
Normalizing flows (Dinh et al., 2015; 2017; Kingma & Dhariwal, 2018) are family of generative models that construct a complex distribution by transforming a simple probability density, such as Gaussian distribution, through sequence of invertible and differentiable mappings. Due to the tractability the exact log-likelihood and the efficiency of the inference and synthesis, normalizing flows have gained popularity in density estimation (Kingma Yang et al., 2019; Prenger et al., 2019; Kumar et al., 2020) and variational inference (Rezende & Mohamed, 2015; Kingma al., 2016). However, for the sake of ensuring the favorable property of closed-fom -ormm density evaluation, the normalizing flows typically require special designs of the sequence of transformations, which, in general, constrain the expressive power of the models.
Dhariwal, 2018: Ho al., 2019;
Energy-based models (EBMs) (Zhu et al., 1998; LeCun et al., 2006; Hinton, 2012) define an un- normalized probability density function of data, which is the exponential of the negative energy function. The energy function is directly defined on data domain, and assigns each input configu- ration with scalar energy value, with lower energy values indicating more likely configurations. Recently, with the energy function parameterized by a modern deep network such as a ConvNet, the ConvNet-EBAe (Xie et al., 2016) have gained unprecedented success in modeling large-scale data sets (Gao et al., 2018; Nijkamp et al., 2019; Du & Mordatch, 2019; Grathwohl al., 2020; Gao et al., 2021; Zhao et al., 2021; Zheng et al., 2021) and exhibited stunning performance in synthesiz- ing different modalities of data, e.g., videos (Xie et al., 2017; 2021c), volumetric shapes (Xie al.,

---

Published as a conference paper at ICLR 2022
201 8b;; 2020b), point clouds (Xie al., 2021a) and molecules (Du et al., 2020). The parameters of the energy function can be trained by maximum likelihood estimation (MLE). However, due to the intractable integral in computing the normalizing constant, the evaluation of the gradient of the log- likelihood typically requires Markov chain Monte Carlo (MCMC) sampling (Barbu & Zhu, 2020), e.g., Langevin dynamics (Neal, 2011), to generate samples from the current model. However, the Langevin sampling on highly multi-modal energy function, due to the use of deep network param- eterization, is generally not mixing. When sampling from a density with multi-modal landscape, the Langevin dynamics, which follows the gradient information, is apt to get trapped by local modes of the density and is unlikely to jump out and explore other isolated modes. Relying on non-mixing MCMC samples, the estimated gradient of the likelihood is biased, and the resulting learned EBM may become an invalid model, which is unable to approximate the data distribution as expected.
Recently, Nijkamp et al. (2019) propose to train an EBM with short-run non-convergent Langevin dynamics, and show that even though the energy function is invalid, the short-run MCMC can be treated as valid flow-like model that generates realistic examples. This not only provides an explanation of why an EBM with a non-convergent MCMC is still capable of synthesizing realistic examples, but also suggests more practical computationally- aaailta affordable way to learn useful genera- tive models under the existing energy-based -bsed frameworks. Although EBMs have been widely applied to different domains, learning short-run MCMC in the context of EBM is still underexplored.
In this paper, we accept the fact that MCMC sampling is not mixing in practice, and decide to give up the goal of training valid EBM of data. Instead, we treat the short-run non-convergent Langevin dynamics, which shares parameters with the energy function, as a flow-like transformation that we call the Langevin flow because it can be considered noise-injected residual network. Even though implementing a short-run Langevin flow is surprisingly simple, which is nothing but a design of a bottom-up ConvNet for the energy function, it might still require a sufficiently large number of Langevin steps (each Langevin step consists of one step of gradient descent and one step of diffu- sion) to construct an effective Langevin flow, so that it can expressive enough to represent the data distribution. Motivated by reducing the number of Langevin steps in the Langevin flow for computational efficiency, we propose the CoopFlow model that trains a Langevin flow jointly with a normalizing flow in a cooperative learning scheme (Xie et al., 2020a), in which the normalizing flow learns to serve as a rapid sampler to initialize the Langevin flow so that the Langevin flow can be shorter, while the Langevin flow teaches the normalizing flow by short-run MCMC transition toward the EBM so that the normalizing flow can accumulate the temporal difference in the transition to provide better initial samples. Compared to the original cooperative learning framework (Xie al., 2020a) that incorporates the MLE algorithm of an EBM and the MLE algorithm of a generator, the CoopFlow benefits from using a normalizing flow instead of generic generator because the MLE of a normalizing flow generator is much more tractable than the MLE of any other generic gener- ator. The latter might resort to either MCMC-based inference to evaluate the posterior distribution or another encoder network for variational inference. Besides, in the CoopFlow, the Langevin flow can overcome the expressivity limitation of the normalizing flow caused by invertibility constraint, which also motivates us to study the CoopFlow model. We further understand the dynamics of coop- erative learning with short-run non-mixing MCMC by information geometry. We provide a justifica- tion that the CoopFlow trained in the context of EBM with non-mixing MCMC is valid generator because it converges to a moment matching estimator. Experiments, including image generation, image reconstruction, and latent space interpolation, are conducted support our justification.
2 CONTRIBUTIONS AND RELATED WORK
Our paper studies amortized sampling for training a short-run non-mixing Langevin sampler toward an EBM. We propose a novel framework, the CoopFlow, which cooperatively trains a short-run Langevin flow as a valid generator and a normalizing flow as an amortized sampler for image rep- resentation. We provide both theoretical and empirical justifications for the proposed CoopFlow algorithm, which has been implemented in the PaddlePaddle deep learning platform. The following some closely related work. We will point out the differences between work and the prior arts to further highlight the contributions and novelty of our paper.
Learning short-run MCMC as generator. Recently, Nijkamp et al. (2019) propose to learn an EBM with short-run con-convergent MCMC that samples from the model, and they eventually keep the short-run MCMC as valid generator and discard the biased EBM. Nijkamp al. (2020b)
2

---

Published as a conference paper at ICLR 2022
use short-run MCMC to sample the latent space of top-down generative model in variational learning framework. An et al. (2021) propose to correct the bias of the short-run MCMC inference by optimal transport in training latent variable models. Pang al. (2020) adopt short-run MCMC to sample from both the EBM prior and the posterior of the latent variables. Our work studies learning normalizing flow to amortize the sampling cost of a short-run non-mixing MCMC sampler (i.e., Langevin flow) in data space, which makes further step forward in this underexplored theme.
Cooperative learning with MCMC teaching. Our learning algorithm is inspired by the cooper- ative learning or the CoopNets (Xie et al., 2018a; 2020a), in which an ConyNet-EBM (Xie et al., 2016) and top-down generator (Han et al., 2017) are jointly trained by jump-starting their max- imum learning algorithms. Xie et al. (2021b) replace the generator in the original CoopNets by a variational autoencoder (VAE) (Kingma CoopPlow algorithm is different from the above prior work in the following two aspects. (i) In the idealized long-run mixing MCMC scenario, our algorithm a cooperative learning framework that trains an unbiased EBM and a normalizing flow via MCMC teaching, where updating normalizing flow with a tractable density is more efficient and less biased than updating generic generator via variational inference in Xie et al. (2021b) or MCMC-based inference as in Xie et al. (2020a). (ii) Our paper has a novel emphasis on studying cooperative learning with short-run non-mixing MCMC, which is more practical and common in reality. Our algorithm actually trains a short-run Langevin flow and a normalizing flow together toward a biased EBM for image generation. We use information geometry to understand the learning dynamics and show that the learned two-flow generator (i.c., CoopFlow) is a valid generative model, even though the learned EBM is biased.
Welling, 2014) for efficient inference. The proposed
Joint training of EBM and normalizing flow. There are other two works studying an EBM and a normalizing flow together as well. To avoid MCMC, Gao et al. (2020) propose to train an EBM using noise contrastive estimation (Gutmann & Hyvärinen, 2010), where the noise distribution is a normalizing flow. Nijkamp et al. (2020a) propose to learn an EBM as an exponential tilting of a pre- trained normalizing flow, so that neural transport MCMC sampling in the latent space of the normal- izing flow can mix well. Our paper proposes to train an EBM and normalizing flow via short-run MCMC teaching. More specifically, we focus on short-run non-mixee non-mixing MCMC and treat it as a valid flow-like model (i.e., short-run Langevin flow) that is guided by the EBM. Disregarding the biased EBM, the resulting valid generator is the combination of the short-run Langevin flow and the nor- malizing flow, where the latter serves as rapid initializer of the former. The form of this two-flow generator shares a similar fashion with the Stochastic Normalizing Flow (Wu et al., 2020), which consists of sequence of deterministic invertible transformations and stochastic sampling blocks.
3
TWo FLOW MODELS
3.1 LANGEVIN FLOW
Energy-based model. Let x. E R.P be the observed signal such as an image. An energy-based model defines an unnormalized probability distribution of as follows:
po(x) Z(Ø) exp[fo(x)],
where fo : RDD IR is the negative energy function and defined by bottom-up neural net- work whose parameters are denoted by 0. The normalizing constant or partition function 7(0) = (exp(f((=)ddz= is analytically intractable and difficult to compute due to high dimensionality of x.
Maximum likelihood learning. Suppose we observe unlabeled training examples {x,, i = 1,... n
from unknown data distribution Paata (z), the energy-based model in Eq. (1) can be trained from {æx} by Markov chain Monte Carlo (MCMC)-based maximum likelihood estimation, in which MCMC samples are drawn from the model (x(x) to approximate the gradient of the log- likeliodl function for updating the model parameters 0. Specifically, the logllikclihood is L(Ø) = "2 > log Po (zz). For large n, maximizing (0) is equivalent to minimizing the Kullback-Leibhe (KI.) divergence DKK.. (Pamal (po). The learning gradient is given by
1 n Σ Vofo
1 Σ Vofo( n i=1
L' (() Epu. [ofo(z)] Epa V [ofo(z))
(2)
where the expectations are approximated by averaging over the observed examples {x;} and the synthesized examples (6:) generated from the current model (z:),, respectively.
3

---

Published as a conference paper at ICLR 2022
Langevin dynamics as MCMC. Generating synthesized examples from po(ar) can be accomplished with a gradientt MCMC such as Langevin dynamics, which is applied as follows
82 = = 2 V vafo( + òc;; Jo~ po(x), G ~ ID), t t= 1- T, (3)
where t indexes the Langevin time step, å denotes the Langevin step size, Et is a Brownian motion that explores different modes, and (o is a uniform distribution that initializes MCMC chains.
Langevin flow. As T + 0o and - O, 2T becomes an exact sample from (o(x) under some regularity conditions. However, it impractical to run infinite steps with infinitesimal step size to generate fair examples from the target distribution. Additionally. convergence of MCMC chains many cases hopeless because (o(r) can be very complex and highly multi-modal, then the gradient-based Langevin dynamics has no way to escape from local modes, so that different MCMC chains with different starting points are unable to mix. Let pe (z) be the distribution of ZT. which is the resulting distribution of x after I' steps of Langevin update starting from zo ~ po(x). Due to the fixed Po (ar).. T and 6, the distribution (z) is well defined, which can be implicitly expressed by
Po (z) = (K) = - p=(2)\Co(z]2)dd
(4)
where Ko denotes the transition kernel of 7 steps of Langevin dynamics that samples po- Generally, po(x) is not necessarily equal to po(x). (x(x) is dependent on T' and s, which are omitted in the no- tation for simplicity. The KL-divergence Dx (No [pa(p)p)))))))))))))))))
(=) [x:)] + log Z. The gradient ascent part of Langevin dynamics, 2; 21-1 + 5 02/2V,, fo increases the negative energy fo(x)) by moving x to the local modes of fo()), and the diffusion part de, increases the entropy of (x)) by jumping out of the local modes. According to the law of thermodynamics (Cover& & Thomas, 2006), IK (pe (x) (e)(pp((x)) decreases to zero monotonically as T + co.
entrropy(pa
3.2 NORMALIZING FLOW
Let E R be the latent vector of the same dimensionality as x. A normalizing flow of the form
(5) where (=)=) is a known prior distribution such as Gaussian white noise distribution (N ID), and Ja : RD' - RDD is a mapping that consists of a sequence of L invertible transformations, i.e., g(2) 9L o.... 02 91 whose inversion 2 9n (x) and log-determinants of the Jaco- bians can be computed closed form. & are the parameters of 9a. The mapping is used to trans- form random vector that follows simple distribution go into flexible distribution. Under the change-of-variable law, the resulting random vector x = 9o (=) has a probability density 4o (x) = gi (h)-1). The successive transformations between æ and z can be expressed as a flow z n h1 99, l12 oh, x, where we define == ho and x == hr. for succinetness. Then the determinant becomes det (x)/Ox)] I det(Ohh-1/ The log-likelihoode of datapoint x can be easily computed by
x = (=); 2 ~ (o(=),
(g (z)) deg (x)/Ox)) Let h4
(ōhl-1) ah:
log (x) = log go(2) + Σ log det
66)
With some smart designs of the sequence of transformations Ja { = 1,..., L}, the log- determinant in Eq. (6) can be easily computed, then the normalizing flow (x) can be trained by
maximizing the exact data log-likelihood [(x:) = #/2 Σm log ga (2x) via gradient ascent algorithm.
4
COOPFLOW: COOPERATIVE TRAINING OF TWO FLOWS
4.1
COOPFLOW ALGORITHM
We have moved from trying to use a convergent Langevin dynamics to train valid EBM. Instead, we accept the fact that the short-run non-convergent MCMC inevitable and more affordable in practice, and treats non-convergent short-run Langevin flow as generator and propose to jointly train it with a normalizing flow as a rapid initializer for more efficient generation. The resulting generator called CoopFlow, which consists of both Langevin flow and normalizing flow.
4

---

Published as a conference paper at ICLR 2022
Specifically, at each iteration, for i = 1, ...,., m, we first generate 2; ~ (O, ID). and then transform 2 by a normalizing flow to obtain f; = (x()). Next, starting from each žy. we run Langevin flow (i.e., a finite number of Langevin steps toward an EBM po(z)) to obtain žy. o; are consid- ered synthesized examples that are generated by the CoopPlow model. We then update a of the normalizing flow by treating c; training data, and update 0 of the Langevin flow according to the learning gradient of the EBM, which is computed with the synthesized examples { and the observed examples } Algorithm I presents a description of the proposed CoopFlow algorithm. The advantage of this training scheme is that we only need to minimally modify the existing codes for the MLE training of EBM pe and normalizing flow qa- The probability density of the CoopFlow ((0.a)) (ar) is well defined, which can be implicitly expressed by
((),a)())= (K)) = (ool'")((zzaz)dd
Kg is the transition kernel of the Langevin flow. If we increase the length T of the Langevin flow, T(Q.a) will converge to the EBM pz)).. The network fa(x) in the Langevin flow is scalar valued and is of free form, whereas the network Io (x) the normalizing flow has high-dimensional output and is of severely constrained form. Thus the Langevin flow can potentially provide tighter fit to Paata (I) than the normalizing flow. The Langevin flow may also be potentially more data efficient as it tends have smaller network than the normalizing flow. On the flip side, sampling from the Langevin flow requires multiple iterations, whereas the normalizing flow can synthesize examples via a direct mapping. It is thus desirable train these two flows simultaneously, where the normal- izing flow serves as an approximate sampler to amortize the iterative sampling of the Langevin flow. Meanwhile, the normalizing flow is updated by a temporal difference MCMC teaching provided by the Langevin flow, to further amortize the short-run Langevin flow.
Algorithm CoopFlow Algorithm
Input: (1) Observed images ; (2) Number of Langevin steps T; (3) Langevin step size o; (4) Learning rate My for Langevin flow; (5) Learning rate no for normalizing flow; (6) batch size m. Output: Parameters (0, a}}
1: Randomly initialize 0 and a.
2: repeat
3: Sample observed examples {²: ~ Pdma(())
4: Sample noise examples {2; mm ~ (2),
5:
Starting from 2;, generate á; = (2x) via normalizing flow.
6:
Starting from ă;. a T-step Langevin flow to obtain i; by following Eq. (3).
7:
Given { update a by maximizing 1 >i log qa (7:) with Eq. (6).
8: Given {a::] and ( update by following the gradient V0 in Eq. (2).
9:
until converged
4.2 UNDERSTANDING THE LEARNED TWo FLOWS
Convergence equations. In the traditional contrastive divergence (CD) (Hinton, 2002) algo- rithm, MCMC chains are initialized with observed data so that the CD learning seeks to minimize (Pama (I) [po(z)) 1kx ( (KCoPasa)))) [=)]][p(=)) where (KoPa denotes the marginal distri- bution obtained by running the Markov transition Ke, which specified by the Langevin flow, from the data distribution Paata- In the CoopFlow algorithm, the learning of the EBM (or the Langevin flow model) follows a modified contrastive divergence, where the initial distribution of the Langevin flow is modified to be normalizing flow 9.- Thus, at iteration I, the update of 0 follows the gradient of DKL (Peaml I DKL (KCtt) (lpa) with respect to O. Compared to the traditional CD loss, the modified one replaces Pdata by ga in the second KL divergence term. At iteration t, the update of the normalizing flow 9a follows the gradient of 1xx (Kace (e [[qa) with respect to a. Let (0*, a be fixed point the learning algorithm converges to, then have the following convergence equations
0t arg min DxL (pamall ll (po) DKL (Co- qa- llp)).
a" arg min DxL (Ko. qa- llq.))
Ideal case analysis. In the idcalized scenario where the normalizing flow 4a, has infinite capacity and the Langevin sampling can mix and converge the sampled EBM, Eq. (9) means that qa.
5

---

Published as a conference paper at ICLR 2022
attempts to be the stationary distribution of Ka+, which is pø- because Kg.- is the Markov transition
kernel of a convergent MCMC sampled from pe-. That is, min, Dx (Kgg [1qa) = 0, thus qa- = Po-- With the second term becoming 0 and vanishing, Eq. (8) degrades to ming Dkr. (Pamal ( and thus 0* is the maximum likelihood estimate, On the other hand, the normalizing flow da chases the EBM po toward Piata thus a" is also a maximum likelihood estimate.
Moment matching estimator. In the practical scenario where the Langevin sampling is not mix- ing, the CoopFlow model मे Kgtng (ote) is an interpolation between the learned Gate> and Pact, and it converges to π² = Ke-qor, which is an interpolation between go and ps*. I² is the short-run Langevin flow starting from qa- towards EBM po.. x² is a legitimate generator because Epa.. Vo (z) = E.- [ e (z)] at convergence. That is, '² leads to moment matching in the feature statistics Vo (x). In other words, n' satisfies the above estimating equation.
Understanding via information geometry. Consider a simple EBM with fo(x) = (0, h(x)), where h(x) is the feature statistics. Since fofo(z) = h(x)). the MLE of the EBM PêlE is a moment matching estimator due to Epe.. [xxx]
E EPOALE [+(=)]] The CoopFlow " also converges to moment matching estimator, i.c., Epa.. [h(z)] Eg- [h(=)]. Figure 1 is illustration of model distributions that correspond to different parameters at convergence. We first introduce three familics v0), and A = {qa, va). which are shown by the red, blue and green curves respectively in Figure 1. Ω is the set of distributions that reproduce statistical prop- erty h()) of the data distribution. Obviously, Pe Pdata- and x = Kg- fa- are included in Ω. O is the set of EBMs with different values of 9, thus Pêsaæ: and py- belong to +. Because of the short-run Langevin flow, pg- not valid model that matches the data distribution in terms of h(r), and thus po- is not in Ω. is the set of normalizing flow models with different values of a, thus fa- and Lowns belong to A. The yellow line shows the MCMC trajectory. The solid segment of the yellow line, starting from go to KCp go,, il- lustrates the short-run non- non-mixing MCMC that is initialized by the normalizing flow 9a- in A and arrives at x' = Ka-qa- in Ω. The dotted segment of the yellow line, starting from rr = Ka- in Ω pa- in O, shows the potential long-run MCMC trajectory, which is not really realized because we stop short in MCMC. If we increase the number of steps of short-run Langevin flow, 1xx [TT (T*l[po-) ) will be monotonically decreasing to 0. Though stops midway in the path toward po.: n is still a valid generator because it in 12.
of distributions: Q = {p [h[[h(r)]
pa [(()])) Q = {ax exp(
P8
Ω
Pdata
MLee
π
A
QUMLE
Qa
Figure 1: An illustration of conver-
gence of the CoopFlow algorithm.
Perturbation of MLE. Poan. is the intersection between θ and Ω. It is the projection of Paata onto θ because MIL = arg ming Dxr. (Pamal Po). aoaas is also a projection of Pam onto A because OMLE = arg min, (Paatal Hhaaallgaa). Generally, JOMLE is far from Pdata due to its restricted form net- work, whereas PomE is very close to Paata due to its scalar-valued free form network. Because the short-run MCMC is not mixing, 0* o OMLE and a* o aMLE. 0" and a are perturbations of ÓLLE and OMLE, respectively. PeSLE shown by an empty dot is not attainable unfortunately. We want to point out that, as T goes to infinity, Pø- Poras and qa
QAMLE- Note that Ω θ and A are high-dimensional manifolds instead of ID curves as depicted in Figure/ 1, and x² may be farther away from Panta than Prwn. is. During learning, (.+1) is the projection of Kgtn (200 on A. At con- vergence, go- is the projection of n Ka-qu on A. There is an infinite looping between Jo- and T" = Ko- at convergence of the CoopFlow algorithm, i.e., n² lifts qa- off the ground A, and the projection drops π back to - Although p3- and qa- are biased, they are not wrong. Many useful models and algorithms, e.g., variational autoencoder and contrastive divergence are also biased to MLE, Their learning objectives also follow perturbation of MLE.
EXPERIMENTS
We showcase experiment results on various tasks. We start from a toy example to illustrate the basic idea of the CoopFlow in Section 5.1. We show image generation results in Section 5.2. Sec- tion 5.3 demonstrates the learned CoopFlow is useful for image reconstruction and inpainting, while Section 5.4 shows that the learned latent space is meaningful that it can be used for interpolation.
6

---

Published as a conference paper at ICLR 2022
5.1 ToY EXAMPLE STUDY
We first demonstrate our idea using a two-dimensional toy example where data lie on a spiral. We train CoopPlow models with different lengths of Langevin flows. As shown in Figure 2, the rightmost box shows the results obtained with 10,000 Langevin steps. We can see that both the nor- malizing flow qa and the EBM pe can fit the ground truth density, which is displayed in the red box, perfectly. This validates that, with a sufficiently long Langevin flow, the CoopFlow algorithm can learn both a valid la and valid po- The leftmost green box represents the model trained with 100 Langevin steps. This is typical non-convergent short-run MCMC setting. In this case, neither Ja nor pe valid, but their cooperation is. The short-run Langevin dynamics toward the EBM actually works
flow-like generator that modifies the initial proposal by the normalizing flow. We can see that the samples from qa are not perfect. but after the modification, the samples from T( fit the ground truth distribution very well. The third box shows the results obtained with 500 Langevin steps. This is still a short-run setting, even though uses more Langevin steps. po becomes better than that with 100 steps, but it is still invalid. With increased number of Langevin steps, samples from both da and ((e.o) are improved and comparable to those the long-run setting with 10,000 steps. The results verify that the CoopFlow might learn a biased BEM and biased normalizing flow if the Langevin flow is non-convergent. However, the non-convergent Langevin flow together with the biased normalizing flow can still form a valid generator that synthesizes valid examples.
Distittioon
Nocmaliding Flow Protebiiit &e
EIM Probahiiiy Po
Nemaliiing Fane Padeiiilly
BIM Pmbbabliity Po
Nonmaizing Flow Pursablliy
EBM Prebebility
Semples
Nomzalizing Samples (42)
Coopllone Samples (Agua))
Nomzaliang
Samples (a.)
Coeplloo Samplai(R
Nammlking Samplax (d,)
Sumple (RGuuc)
Data
T= 100
T
T 10000
Figure 2: Learning CoopFlows on two-dimensional data. The ground truth data distribution shown in the red box and the models trained with different Langevin steps are in the three green boxes. In each green box, the first row shows the Iearned distributions of the normalizing flow and the EBM, and the second row shows the samples from the learned normalizing flow and the learned CoopFlow.
5.2 IMAGE GENERATION
We test our model on image datasets for image synthesis. (i) CIFAR-10 (Krizhevsky & Hinton, 2009) is dataset containing 50k training images and 10k testing images in 10 classes; (ii) SVHN (Netzer et al., 2011 1) is dataset containing over 70k training images and over 20k testing images of house numbers; (iii) CelebA (Liu al., 2015) is celebrity facial dataset containing over 200k im- ages. We downsample all the images to the resolution of 32 x 32. For our model, we show results of three different settings. CoopFlow(T=300 denotes the setting where we train normalizing flow and a Langevin flow together from scratch and use 30 Langevin steps. CoopFlow(T=2000 denotes the setting that we increase the number of Langevin steps to 200. In the CoopFlow( (Pre) setting, we first pretrain a normalizing flow from observed data, and then train the CoopFlow with the parameters of the normalizing flow being initialized by the pretrained one. We use a 30-step-long Langevin flow in this setting. For all the three settings, stage for better performance. We show both qualitative results in Figure 3 and quantitative results in Table 1. To caleulate FID Heusel et al. (2017) scores, generate 50,000 samples each dataset. Our models outperform most of the baseline algorithms. We get lower FID scores comparing to the single normalizing flow models and the prior works that jointly train normalizing flow with EBM, c.g., Gao et al. (2020); Nijkamp et al. (2020a). We also achieve comparable results with the state-of-the-art EBMs. We can see using more Langevin steps pretrained normalizing flow can help improve the performance of the CoopFlow. The former enhances the expressive power, while the latter stabilizes the training. More experimental details and results can be found in Appendix.
slightly increase the Langevin step size at the testing

---

Published as a conference paper at ICLR 2022
5
(a) CIFAR-10
(b) SVHN
(c) CelebA
Figure 3: Generated examples (32 32 pixels) by the CoopFlow models trained on the CIFAR- 10,
SVHN and CelebA datasets respectively. Samples are obtained from the setting of CoopFlow(pre.
Models
FID
Models
FID
VAE
VAE (Kingma & Welling, 2014)
78.41
ABP (Hun et al., 2017)
49.71
PixcCCN (Salimans et al. 2017)
65,93
ABP-SRI (Nijkamp et al., 2020b)
35.23
Autoregressive
PixellON (Ostrovski et al., 2018)
49.46
ABP-OT (An et al., 2021)
19.48
VAE (Kingma & Welling, 2014)
46.78
WGAN Gulllaaan et al., 2017)
GAN
SN GAN (Miyato et al., 2018)
21.70
2:VAE (Dai & Wipf, 2019)
42.81
StyleGANZ-ADA (Karras al., 2020)
RAE (Ghosh
2020)
40.02
Glow (Kingma & Dhariwal. 2018)
41.70
NCSN (Song& & Ermon, 2019)
25.32
DCGAN (Rudford et al., 2016)
21.40
Soore-Basod
NCSN-v2 (Song & Ermon, 2020)
31.75
NT-EBM (Nijkamp et al., 2020a)
NCSN++ (Srin al., 2021)
2.20
LP-EBM (Pang et al.. 2020)
29.44
Flow
Glow (Kingma Dhaziwal, 2018)
45,99
EBM-FCE (Gao et al., 2020)
20.19
Residual Flw (Chen et al., 2019)
46.37
CoopFlow(T=30)
18.11
L.PERM (Pang et al., 2020)
70.15
CoopFlow(TT200)
16.97
EBM SR (Nijkamp al., 2019)
44.50
CoopHoow(Pe)
15.32
EBM-IG (Da & Mordatch, 2019)
38.20
Coup VAEBM (Xie al., 2021b)
36.20
(b) SVHN
CoopNets (Xie ol.,, 2020s)
33.61
BBM
Divergence Triangle (Ila et al., 2020)
30.10
Models
FID
VARA (Grathwohl et al., 2021)
27.50
EBM-CD (Du al., 2021)
25.10
ABP (Han et al., 2017)
GEBM (Arbel et al., 202))
19.31
ABP-SRI (Nijkamp et al., 2020b)
36.84
CF-EBM Zhao al. (2021)
16.71
VAE (Kingma & Welling. 2014)
38.76
VAFBM (Xiao al., 2021)
12.16
Glow (Kingma Dhariwal, 201 8)
23.32
EBM-Diffusion (Gao al. 2021)
9.60
DCGAN (Radford et al., 2016)
12.50
Flow+EBM
NT- (Nijkamp el al., 2020a)
EBM-FCE (Goo et 2020)
37.30
78.12
GEBM (Arbel et al., 2021)
EBM-FCE (Gao et al., 2020)
12.21
5.21
CoopFlow(TT33)
21.16
CoopHoo((T=30)
Ours
CoupFlow(T=200)
18.89
CoopFlow(T=200)
4.90
CoopFlow((Pe)
15.80
CoopFlow(Pre)
4.15
(a) CIFAR-10
(c) CelebA
Table 1: FID scores on CIFAR-1O, SVHN and CelebA datasets. Images are resized to 32 x 32 pixels.
5.3 IMAGE RECONSTRUCTION AND INPAINTING
We show that the learned CoopFlow model is able to reconstruct observed images. We may consider the CoopFlow model ((..a) (a:) latent variable generative model: 2 {o(2); ga (2); e) where denotes the latent variables, e denotes all the injected noises in the Langevin flow, and Fe denotes the mapping realized by a T-step Langevin flow that is actually a T-layer noise-injected residual network. Since the Langevin flow not mixing, x' is dependent on á in the Langevin flow, thus also dependent on 2. The CoopFlow is generator = (2) e). so we can reconstruct any x by inferring the corresponding latent variables 2 using gradient descent on L(z) = || Fo(o. (z), e), with being initialized by 4o- However, g is an invertible transformation, so we can infer z by an efficient way, i.c., we first find & by gradient descent on L($) = ||x Fo e)ll.. with & being initialized by âo (z) where
Po(z) and being set to be 0. and then use 2 = 9- (f) to get the latent variables. These two methods are equivalent, but the latter one is computationally efficient, since computing the gradient on the whole two-flow generator Fo (z), e) is difficult and cine-consuming. Let ă' arg min. L(ā). The reconstruction given by Fo(+"). The optimization is done using 200 steps of gradient descent over á.
8

---

Published as a conference paper at ICLR 2022
The reconstruction results are shown in Figure 4. We use the CIFAR-10 testing set. The right column displays the original images x that need to be recon- structed. The left and the middle columns display D and Fo(#"), respectively. Our model can successfully reconstruct the observed images, verifying that the CoopPlow with a non-mixing MCMC is indeed valid latent variable model.
X Fx(2"
We further show that our model is also capable of doing image inpainting. Similar to image reconstruction, given masked observation Tmask along with a binary matrix M indicating the positions of the unmasked pixels, we op- timize minimize the reconstruction enror between (ă) and mmask in the unmasked area, i.c., L = | @ (Emask - (l)) where @ is the element- still initialized by the normalizing flow. We do experiments on the CelebA training set. In Figure 5, each row shows one example of inpainting with a different initialization of provided by the nor- malizing flow and the first 17 columns display the inpainting results at different optimization iterations. The last two columns show the masked images and the original images respectively. We can see that model can reconstruct the masked areas faithfully and simultaneously fill in the blank areas of the input images. With different initializations, our model inpaint diversified and meaningful patterns.
wise multiplication operator.
Figure 4: Image reconstruction on the CIFAR-I0.
3
7
20
40
60
120
160
200
X
Differest
Figure 5: Image inpainting on the CelebA dataset (32 x 32 pixels). Each row represents one different initialization. The last two columns display the masked and original images respectively. From the 1st column to the 17th column, we show the inpainted images at different optimization iterations.
5.4 INTERPOLATION IN THE LATENT SPACE
The CoopFlow model is capable of doing interpolation the latent space z. Given an image æ, we first find its corresponding 8* using the reconstruction method described in Section 5.3. We then infer by the inversion of the normalizing flow 2 9, (á"). Figure 6 shows two examples of interpolation between two latent vectors inferred from observed images. For each row, the images the two ends are observed. Each image in the middle is obtained by first interpolating the latent vectors of the two end images, and then generating the image using the CoopFlow generator, This experiment shows that the CoopFlow can learn a smooth latent space that traces the data manifold.
Interpolation in latent space
Figure 6: Image interpolation results on the CelebA dataset (32 x 32 pixels). The leftmost and rightmost columns display the images observed. The columns in the middle represent the inter- ploation results between the inferred latent vectors of the two end observed images.
6 CONCLUSION
This paper studies an interesting problem of learning two types of deep flow models in the context of energy-based framework for image representation and generation. One is the normalizing flow that generates synthesized examples by transforming Gaussian noise examples through a sequence of invertible transformations, while the other is the Langevin flow that generates synthesized exam- ples by running non-mixing, con-convergent short-run MCMC toward an EBM. We propose the CoopFlow algorithm to train the short-run Langevin flow model jointly with the normalizing flow serving as a rapid initializer in a cooperative manner. The experiments show that the CoopFlow is a valid generative model that can be useful for image generation, reconstruction, and interpolation.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
The authors sincerely thank Dr. Ying Nian Wu for the helpful discussion on the information geom- etry part and thank Dr. Yifei Xu for his assistance with some extra experiments during the rebuttal. The authors would also like thank the anonymous reviewers of ICLR' 22 program committee for providing constructive comments and suggestions to improve the work.
REFERENCES
Dongsheng An, Jianwen Xie, and Ping Li. Learning deep latent variable models by short-run MCMC inference with optimal transport correction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15415-15424, Virtual Event, 2021.
Michael Arbel, Liang Zhou, and Arthur Gretton. Generalized energy based models. In Proceedings of the 9th International Conference on Learning Representations (ICLR), Virtual Event, 2021.
Adrian Barbu and Song-Chun Zhu. Monte Carlo Methods. Springer Nature, 2020.
Miguel Á. Carreira-Perpiñánn and Geoffrey E. Hinton. On contrastive divergence learning. In Pro-
ceedings of the 10th International Workshop on Artificial Intelligence and Statistics (AISTATS), Bridgetown, Barbados, 2005.
Tian Qi Chen, Jens Behrmann, David Duvenaud, and Jörn-Henrik Jacobsen. Residual flows for in- vertible generative modeling. In Advances in Neural Information Processing Systems (NeurIPS), pp. 9913-9923, Vancouver, Canada, 2019.
Thomas M. Cover and Joy A. Thomas. Elements of information theory, Second Edition. Wiley, 2006.
Bin Dai and David P. Wipf. Diagnosing and enhancing VAE models. In Proceeding of the 7th International Conference on Learning Representations (ICLR), New Orleans, LA, 2019.
Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: non-linear independent components esti-
mation. In Workshop Proceedings of the 3rd International Conference on Learning Representa- lions (ICLR Workshop), San Diego, CA, 2015.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In Proceedings of the 5th International Conference on Learning Representations (ICLR), Toulon, France, 2017.
Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. In Ad- vances in Neural Information Processing Systems (NeurIPS), pp. 3603-3613, Vancouver, Canada, 2019.
Yilun Du, Joshua Meier, Jerry Ma, Rob Fergus, and Alexander Rives. Energy-based models for atomic-resolution protein conformations. In Proceedings of the 8th International Conference on Learning Representations (ICLR), Addis Ababa, Ethiopia, 2020.
Yilun Du Shuang Li, Joshua B. Tenenbaum, and Igor Mordatch. Improved contrastive divergence training of energy-based models. In Proceedings of the 38th International Conference on Machine Learning (ICML), pp. 2837-2848, Virtual Event, 2021.
Ruiqi Gao, Yang Lu, Junpei Zhou, Song-Chun Zhu, and Ying Nian Wu. Learning generative con- vnets via multi-grid modeling and sampling. In Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9155-9164, Salt Lake City, UT, 2018.
Ruiqi Gao, Erik Nijkamp, Diederik P. Kingma, Zhen Xu, Andrew M. Dai, and Ying Nian Wu. Flow contrastive estimation of energy-based models. In Proceedings of the 2020 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7515-7525, Seattle, WA, 2020.
Ruiqi Gao, Yang Song, Ben Poole, Ying Nian Wu, and Diederik P. Kingma. Learning energy energy-based based models by diffusion recovery likelihood. In Proceeding of the 9th International Conference on Learning Representations (ICLR), Virtall Event, 2021.
10

---

Published as a conference paper at ICLR 2022
Partha Ghosh, Mehdi S. M. Sajjadi, Antonio Vergari, Michael J. Black, and Bernhard Schölkopf. the on Learning Representations (ICLR), Addis Ababa, Ethiopia, 2020.
From variational to deterministic autoencoders. In Proceeding the 8th International Conference
Will Grathwohl, Kuan-Chieh Wang, Jörn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. In Proceedings of the 8th International Conference on Learning Representations (ICLR), Addis Ababa, Ethiopia, 2020.
Will Sussman Grathwohl, Jacob Jin Kelly, Milad Hashemi, Mohammad Norouzi, Kevin Swersky, and David Duvenaud. No MCMC for me: Amortized sampling for fast and stable training of energy- based models. In Proceedings of the 9th International Conference Learning Represen- tations (ICLR), Virtual Event, 2021.
Ishaan Gulrajani, Faruk Ahmed, Martín Arjovsky, Vincent Dumoulin, and Aaron C. Courville. Im- proved training of wasserstein gans. In Advances in Neural Information Processing Systems (NIPS), pp. 5767-5777, Long Beach, CA, 2017.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. Proceedings of the Thirteenth International Conference Artificial Intelligence and Statistics (AISTATS), pp. 297-304, Chia Laguna Resort, Sardinia, Italy, 2010.
Tian Han, Yang Lu, Song-Chun Zhu, and Ying Nian Wu, Alternating back-propagation for generator network. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI), pp. 1984, San Francisco, CA, 2017.
Tian Han, Erik Nijkamp, Xiaolin Fang, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Divergence triangle for joint training of generator model, energy-based model, and inferential model. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8670-8679, Long Beach, CA, 2019.
Tian Han, Erik Nijkamp, Linqi Zhou, Bo Pang, Song-Chun Zhu, and Ying Nian Wu. Joint training of variational auto-encoder and latent energy-based model. In Proceedings of the 2020 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7975-7984, Seattle, WA, 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In Advances in Neural Information Processing Systems (NIPS), pp. 6626-6637, Long Beach, CA. 2017.
Geoffrey E. Hinton. Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8):1771-1800, 2002.
Geoffrey E Hinton. A practical guide to training restricted boltzmann machines. In Neural networks: Tricks of the trade, pp. 599-619. Springer, 2012.
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving flow- based generative models with variational dequantization and architecture design. In Proceedings of the 36th International Conference on Machine Learning (ICML), pp. 2722-2730, Long Beach, CA, 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In Advances in Neural Information Processing Systems (NeurIPS), Virtual Event, 2020.
Diederik P. Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR), San Diego, CA, 2015.
Diederik P. Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. In Advances in Neural Information Processing Systems (NeurIPS), pp. 10236-10245, Montréal, Canada, 2018
11

---

Published as a conference paper at ICLR 2022
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Proceedings of the 2nd International Conference on Learning Representations (ICLR), Banff, Canada, 2014.
Diederik P. Kingma, Tim Salimans, Rafal Józefowicz, Xi Chen, llya Sutskever, and Max Welling. Improving variational autoencoders with inverse autoregressive flow. In Advances in Neural In- formatiin Processing Systems (NIPS), pp. 4736-4744, Barcelona, Spain, 2016.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, and Durk Kingma. Videoflow: conditional flow-based model for stochastic video gen- cration. In Proceeding of the 8th International Conference on Learning Representations (ICLR), Addis Ababa, Ethiopia, 2020.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Huang. tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the 2015 International Conference on Computer Vision (ICCV), Santiago, Chile, 2015.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In Proceeding of the 6th International Conference on Learning Representations (ICLR), Vancouver, Canada, 2018.
Radford M Neal. MCMC using hamiltonian dynamics. Handbook of markov chain monte carlo, (11):2, 2011.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Erik Nijkamp. Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Learning non-convergent non- persistent short-run MCMC toward energy-based model. In Advances Neural Information Processing (NeurIPS), pp. 5233-5243, Vancouver, Canada, 2019.
Erik Nijkamp. Ruiqi Gao, Pavel Sountsov, Srinivas Vasudevan, Bo Pang, Song-Chun Zhu, and Ying Nian Wu. Learning energy-based model with flow-based backbone by neural transport MCMC. arXiv preprini (oXXx:::006068977. 2020a.
Erik Nijkamp, Bo Pang, Tian Han, Linqi Zhou, Song-Chun Zhu, and Ying Nian Wu. Learning multi-layer latent variable model via variational optimization of short run MCMC for approximate inference. In Proceedings of the 16th European Conference Computer Vision (ECCV, Part VI), pp. 311-38, Glasgow, UK, 2020b.
Georg Ostrovski, Will Dabney, and Rémi Munos. Autoregressive quantile networks for generative modeling. In Proceedings of the 35th International Conference on Machine Learning (ICML), pp. 3933-3942, Stockholmsmässan, Stockholm, Sweden, 2018.
Bo Pang. Tian Han, Erik Nijkamp, Song-Chun Zhu, and Ying Nian Wu. Learning latent space energy-based prior model. In Advances in Neural Information Processing Systems (NeurIPS), Virtual Event, 2020.
Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: A flow-based generative network for speech synthesis. In Proceeding of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3617-3621, Brighton, United Kingdom, 2019.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In Proceedings of the 4th International Conference on Learning Representations (ICLR), San Juan, Puerto Rico, 2016.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Proceedings of the 32nd International Conference on Machine Learning (ICML) pp. 1530-1538, Lille, France, 2015.
12

---

Published as a conference paper at ICLR 2022
Tim Salimans and Diederik P. Kingma. Weight normalization: A simple reparameterization to celerate training of deep neural networks. In Advances in Neural Information Processing Systems 29 (NIPS), pp. 901, Barcelona, Spain, 2016.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. PixeICNN++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. In Proceeding of the 5th International Conference on Learning Representations (ICLR), Toulon, France, 2017.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems (NeurIPS), pp. 11895-11907, Vancouver, Canada, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. In Advances in Neural Information Processing Systems (NeurIPS), Virtual Event, 2020.
Yang Song, Jascha Sohl- Dickstein,,,, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben
Poole. Score-based generative modeling through stochastic differential equations. In Proceeding of the 9th International Conference on Learning Representations (ICLR), Virtual Event, 2021.
Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. In Advances in Neural Information Processing Systems (NeurIPS), Virtual Event, 2020.
Hao Wu, Jonas Köhler, and Frank Noé. Stochastic normalizing flows. In Advances in Neural Information Processing (NeurIPS), Virtual Event, 2020.
Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat. VAEBM: A symbiosis between varia- tional autoencoders and energy- -based models. In Proceeding of the 9th International Conference on Learning Representations (ICLR), Virtual Event, 2021.
Jianwen Xie, Yang Lu, Song-Chun Zhu, and Ying Nian Wu. theory of generative convnet. In Proceedings of the 33nd International Conference on Machine Learning (ICML), pp. 2635-2644, New York City, NY, 2016.
Jianwen Xie, Cong-Chun Zhu, and Ying Nian Wu. Synthesizing dynamic patterns by spatial- temporal generative convnet. In Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1061-1069, Honolulu, H1, 2017.
Jianwen Xie, Yang Lu, Ruiqi Gao, and Ying Nian Wu. Cooperative learning of energy-based model and latent variable model via MCMC teaching. In Proceedings of the 32nd AAAI Conference Artificial Intelligence, (AAAI), pp. 4292-4301, New Orleans, Louisiana, USA, 2018a.
Jianwen Xie, Zilong Zheng, Ruiqi Gao, Wenguan Wang. Song- Chun Zhu, and Ying Nian Wu. Learn- ing descriptor networks for shape synthesis and analysis. Proceeding of the IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR), pp. 8629-8633 Salt Lake City, UT, 2018b.
Jianwen Xie, Yang Lu, Ruiqi Gao, Song-Chun Zhu, and Ying Nian Wu. Cooperative training of de- scriptor and generator networks. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 42(1):27 45, 2020a.
Jianwen Xie, Zilong Zheng, Ruiqi Gao, Wenguan Wang, Song-Chun Zhu, and Ying Nian Wu. Gen- erative voxelnet: learning energy- based models for 3d shape synthesis and analysis. IEEE Trans- actions on Pattern Analysis and Machine Intelligence (TPAMí), 2020b.
Jianwen Xie, Yifei Xu, Zilong Zheng, Song-Chun Zhu, and Ying Nian Wu. Generative pointnet: Deep energy-based learning on unordered point sets for 3d generation, reconstruction and clas- sification. In Proceeding of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 14976-14985, Virtual Event, 2021a.
Jianwen Xie. Zilong Zheng, and Ping Li. Learning energy based model with variational auto- encoder as amortized sampler. Proceeding of the Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAT), pp. 10441-10451, Virtual Event, 2021b.
13

---

Published as a conference paper at ICLR 2022
Jianwen Xie, Long-Chun Zhu, and Ying Nian Wu. Learning energy- eneryy-based apatial-temport genera- tive convnets for dynamic patterns. JEEE Transactions on Pattern Analysis and Machine Intelli- gence (TPAMI),
2021c.
Guandao Yang. Xun Huang, Zekun Hao, Ming-Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. In Proceeding of the IEEE International Conference on Computer Vision (ICCV). pp. 1540-4549, Scoul, Korea, 2019.
Laurent Younes. On the convergence of markovian stochastic algorithms with rapidly decreasing ergodicity rates. Stochastics: An International Journal of Probability and Stochastic Processes, 65(3-4):177-228, 1999.
Yang Zhao, Jianwen Xie, and Ping Li. Learning energy-based generative models via coarse-to- fine expanding and sampling. In Proceeding of the 9th International Conference Learning Representations (ICLR), Virtual Event, 2021.
Zilong Zheng, Jianwen Xie, and Ping Li. Patchwise generative convnet: Training energy-based models from single natural image for internal learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2961-2970, virtual Event, 2021.
Song-Chun Zhu, Ying Nian Wu, and David Mumford. Filters, random fields and maximum entropy (FRAME): towards a unified theory for texture modeling. International Journal of Computer Vision (IJC)), 27(2):107-12) 1998.
14

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 NETWORK ARCHITECTURE IN THE COOPFLOW
We use the same network architecture for all the experiments. For the normalizing flow ga (z) in our CoopFlow framework, we use the Flow++ (Ho al., 2019) network architecture that was originally designed for the CIFAR-I0 (32 x 32 pixels) dataset in Ho et al. (2019). As to the EBM in our CoopFlow, we use the architecture shown in Table 2 to design the negative energy function fz))..
3 3 Conv2d, str==, pu/==, pule,, ch= Swish
Residual Block, c/=256
3 Conv2d, str= pad=1: Swish
Residual Block, c/=512
3 Conv2d, s/r==, pud=1; Swish Swish
Residual Block, c/24,024
3 x 3 Conv2d, p//=1, Swish (input)
3 3 Conv2d, str=4, pad=0, ch=100; Swish
x 2 Average pooling
Sum over channel dimension
(b) Residual Block architecture
(a) EBM architecture
Table 2: Network architecture of the EBM in the CoopFlow (str: stride, pad: padding, ch: channel).
A.2 EXPERIMENTAL DETAILS
We have three different settings for the CoopFlow model. In the CoopFlow(T=30) setting and CoopFlow([=200) setting, we train both the normalizing flow and the Langevin flow from scratch. The difference between them are only the number of the Langevin steps. The CoopFlow(T=200) uses longer Langevin flow than the CoepFlow(T=3O), We follow Ho et al. (2019) and use the data- dependent parameter initialization method (Salimans & Kingma, 2016) 6) for our normalizing flow in both settings CoopFlow(T=30) (T33)) and ChopFlow(T=2000 On the other hand, setting, we first pretrain normalizing flow on training examples, and then train a 30-step Langevin flow, whose parameters are initialized randomly, together with the pretrained normalizing flow by following Algorithm 1. The cooperation between the pretrained normalizing flow and the untrained Langevin flow would be difficult and unstable because the Langevin flow is not knowledgeable all to teach the normalizing flow. To stabilize the cooperative training and make a smooth transition for the normalizing flow, we include a warm-up phase in the CoopFlow algorithm. During this phase, instead of updating both the normalizing flow and the Langevin flow, we fix the parameters of the pretrained normalizing flow and only update the parameters of the Langevin flow. After a certain number of learning epochs, the Langevin flow may get used to the normalizing flow initialization, and learn to cooperate with Then we begin to update both two flows as described in Algorithm This strategy is effective in preventing the Langevin flow from generating bad synthesized examples at the beginning of the CoopFlow algorithm to ruin the pretrained normalizing flow.
the CoopFlow(P Pre)
We use the Adam optimizer (Kingma & Ba, 2015) for training. We set learning rates no = 0.0001 and 7i = 0.0001 for the normalizing flow and the Langevin flow, respectively. We use B1 = 0.9 and B3 = 0.999 for the normalizing flow and 31 0.5 and 83 0.5 for the Langevin flow. In the Adam optimizer, 31 is the exponential decay rate for the first moment estimates, and 33 is the exponential decay rate for the second-moment estimates. We adopt random horizontal flip data augmentation only for the CIFAR-10 dataset. We remove the noise term in each Langevin update by following Zhao al. (2021). We also propose an alternative strategy gradually decay the effect of the noise term in Section A. The batch sizes for CoopFlow (T=30), CoopFlow(T=200) and CoopFlow(Pre are 28, 32 and 28. The values of other hyperparameters can be found in Table 3.
A.3 ANALYSIS OF HYPERPARAMETERS OF LANGEVIN FLOW
We investigate the influence of the Langevin step size 8 and the number of Langevin steps T on the CIFAR- 10 dataset using the CoopFlow(Pre setting. We first fix the Langevin step size to be 0.03 and vary the number of Langevin steps from 10 to 50. The results are shown in Table 4. On the other hand, we show the influence of the Langevin step size in Table 5, where fix the number
15

---

Published as a conference paper at ICLR 2022
of epochs
# of warm-up epochs for Lang. Flow
# of epochs
# of Langevin steps
Langevin step size (train)
Langevin
Model
Dataset
pretrain
for
step size
Normal. Flow
CoopFlow
(test)
CoopFlow
CIFAR-10
0
0
100
30
0.03
0.04
SVHN
0
0
100
30
0.03
0.035
(T=30)
CelebA
0
0
100
30
0.03
0.035
CoopFlow (T=200)
CIFAR-
0
0
100
200
0.01
0.012
SVHN
0
0
100
200
0.011
0.0125
CelchA
0
0
100
200
0,011
0,013
CoopFlow
CIFAR-10
300
25
75
30
0.03
0.04
SVHN
200
10
30
0.03
0.035
(Pre)
CelebA
30
0.03
0.035
Table 3: Hyperparameter setting in our experiments.
of Langevin steps to be 30 and vary the Langevin step size used in training. When synthesizing examples from the learned models in testing, we slightly increase the Langevin step size by a ratio of 4/3 for better performance. We can see that our choices of 30 the number of Langevin steps and 0.03 as the Langevin step size are reasonable. Increasing the number of Langevin steps can improve the performance in terms of FID, but also be computationally expensive. The choice of = 30 is a trade-off between the synthesis performance and the computation efficiency.
# of Langevin steps
10
20
30
40
50
FID +
16.46
15.20
15.80
16.80
15.64
Table FID scores over the numbers of Langevin steps of CoopFlow(Pre) on the CIFAR-10 dataset.
Langevin step size (train)
0.01
0.02
0.03
0.04
0.05
0.10
Langevin step size (test)
0.013
0.026
0.04
0.053
0.067
0.13
FID +
15.99
16.32
15.80
16.52
18.17
19.82
Table FID scores of CoopFlow(Pre) on the CIFAR-10 dataset under different Langevin step sizes.
A.4 ABLATION STUDY
To show the effeet of the cooperative training, we compare CoopFlow model with an individual normalizing flow and individual Langevin flow. For fair comparison, the normalizing flow com- ponent in the CoopFlow has the same network architecture as that in the individual normalizing flow, while the Langevin flow component in the CoopFlow also uses the same network architecture as that in the individual Langevin flow. We train the individual normalizing flow by following Ho al. (2019) and train the individual Langevin flow by following Nijkamp al. (2019). All three models are trained on the CIFAR-10 dataset. We present a comparison of these three models in terms of FID Table 6, and also show generated samples in Figure 7. From Table 6, we can see that the CoopFlow model outperforms both the normalizing flow and the Langevin flow by large margin, which verifies the effectiveness of the proposed CoopFlow algorithm.
Model
Normalizing flow
Langevin flow
CoopFlow
FID
92.10
49.51
21.16
Table 6: A FID comparison among the normalizing flow, the Langevin flow and the CoopFlow.
A.5 MORE IMAGE GENERATION RESULTS
In Section 5.2, we have shown generated examples from the CoopFlow(Pre) model. In this section, we show examples generated by the CoopFlow(T=30) model in Figure and examples generated by the (oopF)ow((==200) in Figure 9. We see that all the generated examples are meaningful.
16

---

Published as a conference paper at ICLR 2022
(a) Normalizing flow
(b) Langevin flow
(c) CoopFlow(T=30)
Figure 7: Generated examples by (a) the individual normalizing flow, (b) the individual Langevin flow, and (c) the CoopFlow, which are trained on the CIFAR-10 dataset.
On the other hand, we show the examples generated from the normalizing flow component of the CoopPlow model in Figure 10. By comparing the synthesized images shown in Figure and those in Figure 10, we can that there gap between the normalizing flow and the CoopFlow. The samples from the normalizing flow look blurred but become sharp and clear after the Langevin flow revision. This supports our claim in Section 4.2.
15
29
P6
26
(a) CIFAR-10
(b) SVHN
(c) CelebA
Figure 8: Generated examples (32 32 pixels) by the CoopFlow models trained on the CIFAR-10 10,
SVHN and CelebA datasets respectively. Samples are obtained from the CoopFlow(T=30) setting.
00
5
6
77
7
1
0
0
BI
20
55
104
(a) CIFAR-10
(b) SVHN
(c) CelebA
Figure 9: Generated examples (32 32 pixels) by the CoopFlow models trained on the CIFAR-IO.
SVHN and CelebA datasets respectively. Samples are obtained from the CoopFlow(7=200) setting.
A.6 MORE IMAGE INPAINTING RESULTS
We show more image inpainting results in Figure 11. Each panel corresponds to one inpainting task. Each row in each panel represents one inpainting result with different initialization. In each panel, the last two columns show the masked and the ground truth images respectively, and images from the 1tt column the 20th column are the inpainted images different optimization iterations.
17

---

Published as a conference paper at ICLR 2022
(a) CIFAR-10
(b) SVHN
(c) CelebA
Figure 10: Generated examples by the normalizing flow components in the CoopFlow models trained on the CIFAR-IO, SVHN, and CelebA datasets respectively. The CoopFlow models are trained in the CoopFlow(T=30) setting.
0
2
3
7
8
20
40
60
80
100
120
140
160
180 200 Xuszt
Different initiaization
60
80
100
140
160
180
Different iniialiiztiin
Figure 11: More results on image inpainting.
18

---

Published as a conference paper at ICLR 2022
A.7 FID CURVE OVER TRAINING EPOCHS
Figure 12 shows the FID trends during the training of the CoopFlow models on the CIFAR-10 dataset. Each curve represents the FID scores over training epochs. We train the CoopFlow models using the settings of CoopFlow(7=30)) and CoopFlow(pre). For each of them, we can observe that, as the cooperative learning proceeds, the FID keeps decreasing and converges is an empirical evidence to show that the proposed CoopFlow algorithm is convergent algorithm.
low value. This
200
100
CoopFlow (Pre)
CoopFlow
80
FID Scores
150
FID Scores
60
100
50
20
0
20
40
60
80
100
0
20
40
60
80
100
Epoch
Epoch
FID for CoopFlow(7=30).
(b) FID curve for CoopFlow(Pre))
Figure 12: FID curves on the CIFAR-10 dataset. The FID score is reported every 5 epochs.
A.8 QUANTITATIVE RESULTS FOR IMAGE RECONSTRUCTION
We provide additional quantitative results for the image reconstruction experiment in Section 5.3. Following Nijkamp al. (2019), caleulate the per-pixel MSE 1,000 examples in the testing set of the CIFAR-IM 10 dataset. We use a 200-step gradient descent to minimize the reconstruction loss. We plot the reconstruction error curve showing the MSEs over iterations in Figure 13 and report the final per-pixel MSE in Table 7. For baseline, we train an EBM using 100-step short-run MCMC and the resulting model the short-run Langevin flow. We then apply it the reconstruction of the same 1,000 images by following Nijkamp et al. (2019). The baseline EBM has the same network architecture as that of EBM component in our CoopFlow model for fair comparison.
The experiment results show that the CoopFlow works better than the individual short-run Langevin flow in this image reconstruction task.
Model
MSE
Langevin flow / EBM with short-run MCMC
0.1083
CoopFlow (ours)
0.0254
Table 7: Reconstruction error (MSE per pixel).
1.5
Reconstruction Error
1
MSE
0.5
0
50
100
150
200
Iteration
Figure 13: Reconstruction errors (MSE per pixel) over iterations.
19

---

Published as a conference paper at ICLR 2022
A.9 MODEL COMPLEXITY
In Table 8, we present comparison of different models in terms of model size and FID score. Here we mainly compare those models that have a normalizing flow component, c.g., EBM-FCE, NT-EBM, GLOW, Flow++, as well as EBM jointly trained with a VAE generator, e.g., VAEBM. We can see the CoopFlow model has a good balance between model complexity and performance. noteworthy that both the CoopFlow and the EBM-FCE consist of an EBM and a normalizing flow, and also have similar model sizes, but the CoopFlow achieves a much lower FID than the EBM-FCE. Note that the Flow++ baseline uses the same structure as the one in our CoopFlow. By comparing the Flow++ and the CoopFlow, help improve the performance of the normalizing flow in terms FID. On the other hand, although the VAEBM model achieves better FID than ours, but it relies much larger pretrained NVAE model (Vahdat& & Kautz, 2020) that significantly increases the model complexity.
find that recruiting extra Langevin Flow can
Model
# of Parameters
FID |
NT-EBM (Nijkamp et al., 2020a)
23.8M
78.12
GLOW (Kingma & Dhariwal, 2018)
44.2M
45.99
EBM-FCE (Gao et al., 2020)
44.9M
37.30
Flow++ (Ho et al., 2019)
28.8M
92.10
VAEBM (Xiao al., 2021)
135.1M
12.16
(o=3Flow(T=30) (ours)
45.9M
21.16
(T=2Flow(T=200) (ours)
45.9M
18.89
CoopFlow(pre) (ours)
45.9M
15.80
Table 8: A comparison of model sizes and FID scores for different models. FID scores are reported on the CIFAR-10 dataset.
A.10 COMPARISON WITH MODELS USING SHORT-RUN MCMC
Our method is relevant to the short-run MCMC. In this section, we compare the CoopFlow model with other models that use short-run MCMC flow-like generator. The baselines include (1) the single EBM with short-run MCMC starting from the noise distribution (Nijkamp et al., 2019). and (ii cooperative training of an EBM and generic generator (Xie al., 2020a). In Table 9, we report the FID scores of different methods over different numbers of MCMC steps. We can see that with the same number of Langevin steps, the CoopFlow can generate much more realistic image patterns than the other two baselines. The results show that the CoopFlow can use less number of Langevin steps (i.e., a shorter Langevin flow) to achieve better performance.
of MCMC steps
30
40
50
200
Model
Short-run EBM CoopNets
421.3
194.88
117.02
140.79
198.09
54.23
33.74
33.48
34.12
33.85
42.99
38.88
CoopFlow(Pre)
16.46
15.20
15.80
16.80
15.64
17.94
Table 9: A comparison of FID scores of the short-run EBM, the CoopNets and the CoopFlow under different numbers of Langevin steps on the CIFAR-10 dataset.
A.11 NOISE TERM IN THE LANGEVIN DYNAMICS
While for the experiments shown in the main text, we completely disable the noise term de of the Langevin equation presented in Eq. (3) by following Zhao et al. (2021) to achieve better results, here we try an alternative way where we gradually decay the effect of the noise term toward zero during the training process. The decay ratio for the noise term can be computed by the following:
decay ratio = max((1.0 epoch
.0.0)
(10)
K
20

---

Published as a conference paper at ICLR 2022
where K is hyper-paramete controlling the decay speed of the noise term. Such noise decay strategy enables the model to do more exploration in the sampling space at the beginning of the training and then gradually focus on the basins of the reachable local modes for better synthesis quality when the model is about to converge. Note that we only decay the noise term during the training stage and still remove the noise term during the testing stage, including image generation and FID calculation. We carry out experiments on the CIFAR-10 and the SVHN datasets using the CoopFlow(Pre setting. The results are shown in Table 10.
Dataset
K
FID (no noise)
FID (decreasing noise)
CIFAR-10
30
15.80
14.55
SVHN
15
15.32
15.74
Table 10: FID scores for the CoopFlow models trained with gradually reducing noise term in the Langevin dynamics.
A.12 COMPARISON BETWEEN COOPFLOW AND SHORT-RUN EBM VIA INFORMATION GEOMETRY
Figure 14 illustrates the convergences of both the CoopFlow and the EBM with a short-run MCMC starting from an initial noise distribution qo- We define the short-run EBM in more generic form (Xie et al., 2016) as follows
Po (x) = Z(0) explfo(=))qu(x))
(II)
which is an exponential tilting of a known reference distribution (o(x). In general, the reference distribution can be either the Gaussian distribution or the uniform distribution. When the reference distribution is the uniform distribution, 4o can be removed in Eq. (11). Since the initial distribu- tion of the CoopFlow is the Gaussian distribution normalizing flow. For a convenient and fair comparison, we will use the Gaussian distribution as the reference distribution of the EBM in Eq. (11). And the CoopFlow and the baseline short-run EBM will use the same EBM defined in Eq. (11) their frameworks. We will use Pa to denote the baseline suo EBM and keep using pe to denote the EBM in the CoopFlow.
which is actually the prior distribution of the
There are three families of distributions: Ω {p [x()]]]]]]]]]]]]]]]]]]] = Ex [hx)]] C {po(z)
exp( h(z)) yoo (z) v0), and A = {ga, Va), which are shown by the red, blue and green curves respectively Figure 14, which is an extension of Figure by adding the following elements:
go. which is the initial distribution for both the CoopFlow and the short-run EBM. It belongs to θ because it corresponds to = O. qo is just noise distribution thus it is far under the green curve. That is, it is very far from qax because qa. has been already a good approximation of pe-.
ga which is the learned transformation of the normalizing flow 9a, and is visualized as mapping from to 9a- by directed brown line segment.
The MCMC trajectory of the baseline short-run EBM pa. which is shown by the yellow line the right hand side of the blue curve. The solid part of the yellow line, starting from to म Kg- go, shows the short-run non-mixing MCMC starting from the initial in Ω. The dotted part of the yellow line is
Gaussian distribution do in O and arriving the potential long-run run MCMC trajectory that is unrealized.
By comparing the MCMC trajectories of the CoopFlow and the short-run EBM Figure 14, we can find that the CoopFlow has a much shorter MCMC trajectory than that of the short-run EBM, since the normalizing flow ga- in the CoopFlow amortizes the sampling workload for the Langevin flow in the CoopFlow model.
A.13 MORE CONVERGENCE ANALYSIS
The CoopFlow algorithm simply involves two MLE learning algorithms: (i) the MLE learning of the EBM po and (ii) the MLE learning of the normalizing flow qa- The convergence of each of the two
21

---

Published as a conference paper at ICLR 2022
θ
po
Ω
Pe
Pdata
MOMLE
A
QAMLE
qu
flow: 9a.
9o
Figure 14: A comparison between the CoopFlow and the short-run EBM.
learning algorithms has been well studied and verified in the existing literature, e.g., Younes (1999); Xie et al. (2016); Kingma & Dhariwal (2018). That is, each them has a fixed point. The only interaction between these two MLE algorithms in the proposed CoopFlow algorithm is that, in each learning iteration, they feed each other with their synthesized examples and use the cooperatively synthesized examples in their parameter update formulas. To be specific, the normalizing flow uses its synthesized examples to initialize the MCMC of the EBM, while the EBM feeds the normaliz- ing flow with its synthesized examples training examples. The synthesized examples from the Langevin flow are considered the cooperatively synthesized examples by the two models, and are used compute their learning gradients. Unlike other amortized sampling methods (e.g.. Han al. (2019); Grathwohl et al. (2021)) that uses variational learning, the EBM and normalizing flow in our framework do not back-propages
each other through the cooperatively synthesized examples. They just feed each other with some input data for their own training algorithms. That is, each learning algorithm will still converge to fixed point.
Now let us analyze the convergence of the whole CoopFlow algorithm that alternates two maximum likelihood learning algorithms. We first analyze the convergence of the objective function at each learning step, and then we conclude the convergence of the whole algorithm.
The convergence of CD learning of EBM. The learning objective of the EBM is to minimize the KL divergence between the EBM pe and the data distribution Pdata Since the MCMC of the EBM our model is initialized by the normalizing flow la. follows modified contrastive divergence algorithm. That is, iteration t, it has the following objective,
((++1) arg min DxL (Pdatal llpe) DKL (accooacollppp)
(12)
No matter what kind of distribution is used to initialize the MCMC, it will has a fixed point when the learning gradient of 0 equals to 0, i.c., L'(0) Σ Vefe(z.)) 1 Σ Vefe(ā.) 0. The initialization the MCMC only affects the location of the fixed point of the learning algorithm. The convergence and the analysis of the fixed point of the contrastive divergence algorithm has been studied by Hinton (2002); Carreira- Perpiina & Hinton (2005).
The convergence of MLE learning of normalizing flow. The objective of the normalizing flow to learn to minimize the KL divergence between the normalizing flow and the Langevin flow (or the EBM) because, in each learning iteration, the normalizing flow uses the synthesized examples
22

---

The convergence of CoopFlow. The CoopFlow alternates the above two learning algorithms. The EBM learning seeks to reduce the KI. divergence between the EBM and the data, i.e., Po -> Pdata; while the MLE learning of normalizing flow seeks to reduce the KI. divergence between the normalizing flow and the EBM, i.e., do -> Pa- Therefore, the normalizing flow will chase the EBM toward the data distribution gradually. Because the process pa Pdata will stop at fixed point, therefore Qa + pa will also stop at fixed point obviously. Such a chasing game a contraction algorithm, therefore the fixed point of the CoopFlow exists. Empirical evidence also support Published as a conference paper at ICLR 2022
generated from the Langevin dynamics training data. At iteration t, it has the following objective
(1+1) arg min DEL (Kgrogaoll4..)
(13)
which is a convergent algorithm each The convergence has been studied by Younes (1999).
claim. we use (0", a²) ) to denote the fixed point of the CoopFlow, according to the definition of fixed point, (0", .a") will satisfy
0r = arg min DxL (Pámal l|po) -DKL (Ko--a- (pp).
(14)
o= arg min DxL (Koo qa 11
(15)
The convergence of the cooperative learning framework (CoopNets) that integrates the MLE algo- rithm of an EBM and the MLE algorithm generic generator has been verified and discussed in Xie et al. (2020a). The CoopFlow that uses normalizing flow instead of a generic generator has the same convergence property as that of the original CoopNets. The major contribution in our paper is start from the above fixed point equation analyze where the fixed point will be our learning algorithm, especially when the MCMC is non-mixing and non-convergent. This goes beyond all the prior works about cooperative learning.
23


Reviews:
[{"review_id": "xR0BwH8hG0", "reviewer": "Reviewer_1uN8", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper provides a novel cooperative way to train the flow model and energy model. The idea is very attractive, and could be able to solve existing difficulties of generative models. My major concern for the CoopFlow comes from the fact that the training procedure cannot be interpreted as the minimization (or minimax) of a loss function. It is not clear whether the cooperative learning process is always convergent.\n\nSome other comments are as follows:\n\n1) The Langevin dynamics used in the paper is not an exact MCMC chain due to the time discretization. Is it possible to utitlize some MCMC samplers, e.g., Metropolis-adjusted Langevin algorithm in CoopFlow?\n\n2) It is claimed that the KL div between \\tilde p_\\theta and p_\\theta \"decreases to zero monotonically\". This conclusion is not widely known and requires reference.\n\n3) Some important previous models and related references are omitted, including the deep energy model proposed by Liu within the stein variational framework, and the stochastic NF proposed by Noe consisiting of NF and MCMC.\n\n4) Analysis in Section 4.2 is too confusing for the reader. This section is called \"theoretical understanding\" and there is no theorem. Can authors provide more understandable explanations?", "summary_of_the_paper": "This paper develops a new generative model, called CoopFlow, by combining the energy-based model (EBM), normalizing flow (NF) and Langevin dynamics (LD). The CoopFlow updates all the parts in an cooperative manner: Parameters of NF are updated by using samples modified by LD, LD is performed based on the potential function provided by the EBM, and the gradient of log-likelihood of EBM are estimated according to samples generated by NF + LD. Here, LD can overcome the limitation of the model capacity of NF, and NF + LD can provide a more accurate estimation of the gradient of EBM loss. After learning, a powerful sampler and an accurate EBM can be obtained.", "summary_of_the_review": "The article not only introduces some innovations, but also conducts a large number of experiments to demonstrate the superiority of the algorithm.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dq_DxuSZ4Ln", "reviewer": "Reviewer_bAHo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrength\n- The idea of the paper is well simple, and it seems to work well to generate realistic samples. \n- It makes progress with respect to Cooperative learning via MCMC teaching, by using the normalizing flow instead of other kind of generators. \n\nWeakness\n- The theoretical validity of the generator is not very clear based on Section 4.2. In particular, there is no condition  or evidence to show that the generator converges to a fixed point in (8) and (9). Why there exists a fixed point? \n- If possible, the baseline in Table 1 should also include the result of Ho2019 (used in the proposed model) in the category of Flow, to see if the MCMC teaching is important. \n\nQuestions: \n- It is not clear what is evaluated in Section 5, is it the training set or test set? Can MSE score (often used in image reconstruction) be provided for Figure 3? \n- What would happen if you use 200-step  instead of 30-step in CoopFlow (Pre) in section 5.2?\n\nTypo\n- section 4.1, in practise -> in practice", "summary_of_the_paper": "This paper proposes to train an energy-based model with a normalizing flow to achieve rapid and high-quality MCMC sampling. \nIt shows the trained CoopFlow is capable of synthesizing toy data and realistic images (reasonable compared to many baseline methods in terms of common evaluation metrics).  Applications to image reconstruction and interpolation are also included. ", "summary_of_the_review": "This paper introduced a simple idea to train an energy-based model, which is very challenging in high-dimensions. The numerical result makes quite significant progress over state-of-the-art methods in terms of FID scores on natural images.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7b5UVrwcHys", "reviewer": "Reviewer_QoKP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper describes a reasonable approach to generative modeling and demonstrates promising metric values. However, the main issue of the paper is that the particular choice of the generative models is not well-motivated. That is, the first question that appears in the reader's mind is why do we need to combine a flow with a short-run chain? Couldn't we use any other pair of $C_n^2$ combinations (where $n$ is the number of different approaches) to learn an efficient generative model? Finally, couldn't we just increase the size of our baseline model instead of mixing it with another approach? Unfortunately, the paper doesn't address any of these questions. To be more precise, the authors do not provide any motivation for the usage of these particular models. The main argument in favor of the proposed approach is the empirical comparison, which is, however, not extensive, since the models are not compared in terms of computational budget.\n\nOther comments:\n- The proposed objective in equation (8) closely resembles the Contrastive Divergence objective as noted by the authors. The contrastive divergence objective is notorious for its biasedness since the samples from the model ($K_\\theta q_\\alpha$ here) are usually treated to be independent of parameters $\\theta$. I wonder if the authors follow the same way for their objective. Unfortunately, the authors neither clarify this in the paper nor provide an analysis of the bias (in the case of the biased scheme).\n- The ability to model the density is not demonstrated empirically. The authors mention that their approach is able to model the density of the learned distribution (which could be a very nice motivation by the way), but they do not validate this property empirically even on a toy example.\n- There is no convergence analysis. The authors neither prove nor demonstrate empirically the convergence of the procedure. Moreover, according to the last paragraph of Section 4.2, the scheme performs \"an infinite looping\" hence does not converge. This non-convergence property is usually considered as a significant flaw of the approach (e.g. in GANs literature).\n- Some empirical results are not trustworthy. Providing comparison with other generative models the authors report numbers from different papers. However, the comparison on Celeba is hardened by the fact that many papers do not downsample the images to 32x32 resolution as the authors do. For instance, NT-EBM (Nijkamp, 2020) uses 64x64 resolution and couldn't be compared in terms of FID to the proposed model.\n- Reconstruction use case is not clear. The experiments with image reconstruction require additional clarification. It is not clear what kind of a problem the authors approach there. Are these images corrupted by some noise? The choice of the intermediate representation (the output space of the flow) for the latent space is also not motivated. Why couldn't one take the initial distribution that is an input for the flow?\n- Overall, the numerous applications listed after the generative modeling experiments should be measured quantitatively rather than qualitatively by several examples.", "summary_of_the_paper": "The paper proposes a generative model that consists of two components: a normalizing flow and an energy-based model with short-run MCMC as proposed in (Nijkamp, 2019). The whole procedure operates by pushing some simple distribution through the NF and then running the short-run chain on these samples. Both models are learned jointly as follows. The normalizing flow tries to capture the limiting distribution of the EBM by minimizing \n$$\\text{KL}(K_\\theta q_\\alpha \\Vert q_\\alpha),$$ \nwhere $q_\\alpha$ is the density of the NF with parameters $\\alpha$ and $K_\\theta$ is the transition kernel of the short-run chain with parameters $\\theta$. Simultaneously, the EBM is learned to capture the data by minimizing \n$$\\text{KL}(p_{\\text{data}}\\Vert p_\\theta) - \\text{KL}(K_\\theta q_\\alpha\\Vert p_\\theta).$$\n\nIn Section 4.2, the authors provide some analysis of the proposed procedure. Namely, the authors consider the perfect scenario (where the NF matches the EBM density and they both match data) and give some intuition for the case when the models are not expressive enough to perfectly match the data.\n\nFinally, the authors provide an empirical study of the proposed procedure. They report metrics for image generation on CIFAR10, SVHN, Celeba (downsampled to 32x32); give several examples of image inpainting and latent space interpolation.", "summary_of_the_review": "The paper describes a reasonable approach to generative modeling and demonstrates promising metric values. However, the approach itself is a combination of two well-known models. The main issue of the paper is that the particular choice of the generative models is not well-motivated.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4GgUkv5FLJt", "reviewer": "Reviewer_DUki", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is strongly motivated by the short-run MCMC energy-based model [1]. The key difference between this work and [1] is the use of a normalizing flow model for a nice initialization to shorten MCMC steps even more. \n\nThe idea is sound and well-motivated, however, my main concern on this work is that it seems both the theoretical explanation and empirical results do not support such a motivation very clearly. \n\n* First, the theoretical explanation reveals that the short-run MCMC CoopFlow $\\pi^*$ is a moment matching estimator of $p_{data}$. However, the current version of the explanation does not clearly show that $q_{\\alpha}$ initialization can yield a shorter transition path (i.e., lower MCMC steps) compared to the standard Langevin flow started from $p_{0}$ (which is the main motivation of this work). Of course, it can be deduced from the fact that $q_{\\alpha}$ tries to mimic $\\pi$ thus automatically $D(\\pi, q_{\\alpha})$ can be lower than $D(\\pi, p_{0})$ for a certain discrepancy D in most cases; but it will be nice if the authors can discuss such a case in a more systematic way, with adding some intuitive comparison of Figure 1 with Figure 5 of [1].\n\n* In addition, the current empirical results do not validate the authors’ motivation sufficiently well, because a detailed cost-vs-performance trade-off study is not presented here. Table 1 (a) only reveals that the proposed CoopFlow gets worse FID scores than, for example, NCSN++ or EBM-Diffusion. I don't think every newly proposed model should beat the best model. However, since the main advantage of CoopFlow is a fast generation (as well as training) owing to the short-run MCMC with the normalizing flow-based amortized sampler, the author should discuss such a cost perspective in detail. For example, how many iteration MCMC/SDE steps are required for other models, and how about generating samples with the same (and small) number of steps for all models? Currently, I can only conclude the CoopFlow outperforms each of its components (it is also not absolutely certain because the reference flow model, Flow++, seems to be not evaluated solely, thus the author should ), which is also a nice contribution but not greatly fascinating.\n\n***\n\n[1] Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Learning non-convergent nonpersistent short-run MCMC toward energy-based model. arXiv preprint arXiv:1904.09770, 2019.\n", "summary_of_the_paper": "The authors propose a cooperative learning framework of normalizing flow and energy-based Langevin MCMC models (CoopFlow). Specifically, the normalizing flow suggests a better initialization for the Langevin flow, then the latter generates synthetic samples via short-run MCMC. While the Langevin flow is trained with a standard MCMC likelihood toward the data distribution, the normalizing flow chases the Langevin one by maximizing the likelihood of the synthetic samples. The proposed CoopFlow seems to outperform each of its components.", "summary_of_the_review": "Overall, while the paper is interesting, the lack of justification for the authors’ motivation prevents me from recommending a clear acceptance for this paper. My current evaluation of this paper is borderline but can be changed after reading the author’s rebuttal.\n\n***\nPost-rebuttal: I appreciate the authors’ thorough responses to my questions. After reading the responses, I have tended to accept this work, thus raised the review score accordingly. However, this rating is only marginally above the borderline, thus I would not champion the paper for acceptance.\n\nAfter the authors' follow-up response: I feel more comfortable accepting this work. I am sorry to keep my score as 6, mainly due to there is no review score of 7 in the system. Instead, I increased the significance score of this paper accordingly.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a methodological improvement in the Langevin-based training of energy-based models. The idea is to initialize the Langevin flow used to train an energy-based model with a normalizing flow which learns to mimic the Langevin flow as the energy-based model is being trained. The method is empirically evaluated on synthetic data and image benchmarks.

The reviewers are currently divided: one argues for strong rejection, two for weak acceptance, and one for strong acceptance. In summary, the reviewers have expressed two main points of criticism: (a) that the motivation is unclear or not experimentally demonstrated; (b) that the convergence properties of the algorithm are unclear. Regarding (a), I believe the authors have adequately addressed this concern, and in my judgement the method is sufficiently motivated. Regarding (b), the authors responses have mostly relied on non-rigorous argumentation and appeal to prior work, so I don't think the issue has been addressed to the reviewers' satisfaction. Having said that, in my judgement lack of clarity regarding convergence is not a sufficient reason to reject the paper, as there don't seem to be reasonable doubts that the method doesn't converge in practice.

On balance, although the paper has certain weaknesses, it proposes an interesting and potentially useful method without major technical inadequacies, so I'm leaning towards recommending acceptance.
Decision: Accept (Poster)