Pseudo-Spherical Contrastive Divergence
Authors: Lantao Yu
Jiaming Song
Yang Song
Stefano Ermon
Venue: NeurIPS 2021 Poster (2021)
Keywords: energy-based models
proper scoring rules
deep generative models

Abstract: Energy-based models (EBMs) offer flexible distribution parametrization. However, due to the intractable partition function, they are typically trained via contrastive divergence for maximum likelihood estimation. In this paper, we propose pseudo-spherical contrastive divergence (PS-CD) to generalize maximum likelihood learning of EBMs. PS-CD is derived from the maximization of a family of strictly proper homogeneous scoring rules, which avoids the computation of the intractable partition function and provides a generalized family of learning objectives that include contrastive divergence as a special case. Moreover, PS-CD allows us to flexibly choose various learning objectives to train EBMs without additional  computational cost or variational minimax optimization. Theoretical analysis on the proposed method and extensive experiments on both synthetic data and commonly used image datasets demonstrate the effectiveness and modeling flexibility of PS-CD, as well as its robustness to data contamination, thus showing its superiority over maximum likelihood and $f$-EBMs.

Pseudo-Spherical Contrastive Divergence
Lantao Yu
Jiaming Song
Computer Science Department
Computer Science Department Stanford University tsongles... stanford
Stanford University
lantaoyu@cs sww.sstd..deeeeo.mmmm edu
Yang Song
Stefano Ermon
Computer Science Department
Computer Science Department Stanford University ermon0ca. stanford. edu
Stanford University
yangsongocs stanford. edu
Abstract
Energy-based models (EBMs) offer flexible distribution parametrization. How- ever, due to the intractable partition function, they are typically trained via con- trastive divergence for maximum likelihood estimation. In this paper, we propose pseudo-spherical contrastive divergence (PS-CD) to generalize maximum likeli- hood learning of EBMs. PS-CD is derived from the maximization of family of strictly proper homogeneous scoring rules, which avoids the computation of the intractable partition function and provides a generalized family of learning objectives that include contrastive divergence as a special case. Moreover, PS-CD allows us to flexibly choose various learning objectives to train EBMs without additional computational cost or variational minimax optimization. Theoretical analysis the proposed method and extensive experiments on both synthetic data and commonly used image datasets demonstrate the effectiveness and modeling flexibility of PS-CD, well its robustness to data contamination, thus showing its superiority over maximum likelihood and f-EBMs.
Introduction
Energy-based models (EBMs) provide unified framework for generative and discriminative learning by capturing dependencies between random variables with an energy function. Due to the absence of the normalization constraint, EBMs offer much more flexibility in distribution parametrization and architecture design compared to properly normalized probabilistic models such as autoregressive
models [53 23 flow-based models [14 1S 46) and sum-product networks [72]. Recently, deep
EBMs have achieved considerable success in realistic image generation [7
66 12
30 molecular
modeling (90) and model-based planning [16], thanks to modern deep neural networks
54 49
for parametrizing expressive energy functions and improved Markov Chain Monte Carlo (MCMC)
techniques 62 75 40
I7 [6) for efficiently sampling from EBMs.
Training EBMs consists of finding an energy function that assigns low energies to correct config- urations of variables and high energies incorrect ones (55). where central concept is the loss functional that is used to measure the quality of the energy function and is minimized during training. The flexibility of EBMs does not come for free: it makes the design of loss functionals particularly challenging, as it usually involves the partition function that is generally intractable to compute. As result, EBMs are typically trained via CD (37] which belongs the "analysis synthesis" scheme
31 and performs sumpling-based based estimation of the gradient of KL between data distribution and energy-based distribution. Since different loss functionals will induce different solutions in practice (when the model is mie-spicd and data is finite) and KL may not provide the right inductive
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

inspired by the great success of implicit generative models (28 67 31, 89 proposed a variational framework to train EBMs by minimizing general f-divergences [TO]. Although this framework enables us to specify various modeling preferences such as diversity/quality tradeoff, they rely on learning additional components (variational functions) within minimax framework, where the optimization is complicated by the notion of Nash equilibrium and local optimality [2 and suffers from instability and con-convereence issues [59]. Along this line, noise contrastive estimation (NCE) (34] can train EBMs with a family of loss functionals induced by different Bregman divergences. However, in practice, NCE usually relies on carefully-designed noise distribution such as context-dependent noise distribution 4T or joint learning of flow- based noise distribution [21.
bias (25 91
In this paper, we draw inspiration from statistical decision theory [3] and propose a novel perspective for designing loss functionals for training EBMs without involving auxiliary models or variational optimization. Specifically, to generalize maximum likelihood training of EBMs, we focus on maximizing pseudo-spherical scoring rules 76, 27). which are strictly proper such that the data distribution the unique optimum and homogeneous such that they can be evaluated without the knowledge of the normalization constant. Under the "analysis by synthesis" scheme used in CD and f-EBM [89], we then derive a practical algorithm termed Eseudo-Spherica Contrastive Divergence (PS-CD). Different from f-EBM, PS-CD enables us to specify flexible modeling preferences without requiring additional computational cost or unstable minimax optimization. We provide a theoretical analysis on the sample complexity and convergence property of PS-CD, as well as its connections to maximum likelihood training. With experiments on both synthetic data and commonly used image datasets, show that PS-CD achieves significant sample quality improvement over conventional maximum likelihood training and competitive performance to f-EBM without expensive variational optimization. Based on a set of recently proposed generative model evaluation metrics (61), we further demonstrate the various modeling tradeoffs enabled by PS-CD, justifying its modeling flexibility. Moreover, PS-CD also much more robust than CD in face of data contamination.
2 Preliminaries
2.1 Energy-Based Distribution Representation and Sampling
Given a set of i..d. samples ( from some unknown data distribution p(æ)) defined over the sample space X' C TR.", the goal of generative modeling to learn a 0-parametrized probability distribution qo (x()) to approximate the data distribution p(z). In the context of energy-based mod- cling, instead of directly parametrizing a properly normalized distribution, we first parametrize an unnormalized energy function Es X + IR, which further defines normalized probability density via the Boltzmann distribution:
(z)) = (a( Zo
exp( (FF(-F(())
(1)
Zo
where Zo = z===
is the partition function (normalization constant). In this paper, unless otherwise stated, we will use to denote an unnormalized density and q to denote the corresponding normalized distribution. We also assume that the exponential of the negative energy belongs to the L space, E := {Ee X + R: fx (()(-Ee(x))d < oo). i.e., Zo is finite.
Since energy-based models (EBMs) represent a probability distribution by assigning unnormalized scalar values (energies) to the data points, we can use any model architecture that outputs a bounded real number given an input to implement the energy function, which allows extreme flexibility in distribution parametrization. However, it is non-trivial to sample from an EBM, usually requiring MCMC (75) techniques. Specifically, this work we consider using Langevin dynamics [62 [8), a gradient- gradient-based based MCMC method that performs noisy gradient descent to traverse the energy landscape and arrive at the low-energy configurations:
3; = ..-
C - V-
Vezi,
(2)
where 2, ~ I). The distribution of क7 converges to the model distribution (0(x) x exp(-Eo(x)) when + 0 and T - oo under some regularity conditions (88). In order to sample from an energy-based distribution efficiently, many scalable techniques have been proposed such as learning non-convergent, porsisssit,,nt, short-run MCMC (66] and using a sample replay buffer to improve mixing time and sample diversity [17. In this work, we leverage these recent advances when we need to obtain samples from an EBM.

---

2.2 Maximum Likelihood Training of EBMs via Contrastive Divergence
The predominant approach to training explicit density generative models is to approximately minimize the KL divergence between the (empirical) data distribution and model distribution. Minimizing KL divergence is equivalent the following maximum likelihood estimation (MLE) objective:
min LIE p) = min - -p(=)) [log qe (x)] = min pp()) [E [x)] + log Ze.
(3)
0
Because of the intractable partition function (an integral over the sample space), we cannot directly optimize the above MLE objective. To tackle this issue, [37] proposed contrastive divergence (CD) algorithm as a convenient way to estimate the gradient of CMLE ( p) using samples from gø:
VoLMI. (o.,).(OOp)) Ep(a)) V Exr)) Ve log Zo pp)() ) Vo Eg (a:) Ey [ ( (4)
which can be interpreted as decreasing the energies real data from p and increasing the energies of fake data generated by go. As diseussed above, evaluating Equation (4 typically relies MCMC
methods such as the Langevin dynamics sampling procedure defined in Equation
(2) to produce
samples from the model distribution ge. which induces surrogate gradient estimation:
foLcD-x(O;p)) = Ep(z) V Ea [=]] gg (w) V EaE)]]
(5)
where o denotes the distribution after K steps of MCMC transitions from an initial distribution (typically data distribution or uniform distribution), and Equation 4 corresponds to LCD
2.3 Strictly Proper Scoring Rules
Stemming from statistical decision theory 13 scoring rules evaluate the quality of probabilistic forecasts by assigning numerical scores based on the predictive distributions and the events that materialize. Formally, consider compact sample space X. Let M be a space of all locally 1- integrable non-negative finite measures and P be a subspace consisting of all probability measures on the sample space X. A scoring rule S(x, q) specifies the utility of forecasting using a probability forecast q E 'P for a given sample x E X. With slightly abused notation, we write the expected score of ((x, q) under reference distribution p as:
Sp. q) == Pptt) (S 4)}-
(6)
Definition 1 (Proper Scoring Rules (6)). A scoring rule S : X P - R is called proper relative to P if the corresponding expected score satisfies:
V E S(p(()))) 9) S S(p,p).
It is strictly proper if the equality holds if and only if =
In prediction and elicitation problems, strictly proper scoring rules encourage the forecaster to make honest predictions based on their true beliefs [22]. In estimation problems, where we want to approximate distribution p with another parametric distribution go, strictly proper scoring rules provide attractive learning objectives:
arg max S(p, ge) arg max En(a)) p (when p PP)). EP
Po
When a scoring rule S is strictly proper relative to P. the associated generalized entropy function and divergence function are defined as:
G(p) == sup S(p. g) S(p.p), D(p.g) = :(pp: - S(p. q) 2 0. 9EP
(9)
G(p) is convex and represents the maximally achievable utility, while D q) is the Bregman divergence (6] associated with the convex function G and the equality holds only when p = 4.
Next, we introduce a specific kind of scoring rules that are particularly suitable for learning unnor- malized statistical models.
Definition 2 (Homogeneous Scoring Rules (69]). A scoring rule is homogeneous if it satisfies (here the domain of the score function is extended to x M):
AA > 0, X, S(x.9) S(x, A Ag).
(10)
3

---

Since scaling the model distribution q by positive constant À does not change the value of a homo- geneous scoring rule, such homogeneity allows to evaluate without computing the intractable partition function of an energy-based distribution. Thus, strictly proper and homogeneous scoring rules are natural candidates for new training objectives of EBMs.
Example 1. A notable example of scoring rules is the widely used logarithm score: S(a,
log ((z). The associated generalized entropy is the negative Shannon entropy: G(p) =
Eptze [log p(a:), and the associated Bregman divergence Bytz) [log(p(>)/q(<)))
the KI. divergence: D(p:q)
From Definitions [ we know that the logarithum score is strictly proper but not homogeneous. Specifically, for --parumetrized energy-bused distribution go 9 since S(x, S(x, Zo (qa) S(x (0) + log Zo + S(x, qo) and log Zo cannot be ignored during the optimization of O, need to use tailored methods such contrastive divergence [37] or doubly dual embedding [11] to tackle the intractable partition function.
Training EBMs by Maximizing Homogeneous Scoring Rules
In this section, we derive new principle for training EBMs from the perspective of optimizing strictly proper homogeneous scoring rules. All proof' for this section can be found in Appendix
3.1 Pseudo-Spherical Scoring Rule
In this section, we introduce the pseudo-sphericee scoring rule, which is representative family of strictly proper homogeneous scoring rules that have great potentials for training deep energy-bases models and allow flexible and convenient specification of modeling preferences, yet have not been explored before in the context of energy-based generative modeling.
Definition 3 (Pseudo-Spherical Scoring Rule rule is defined as:
76 27 ) For 7 O, the pseudo-spherical scoring
((x)
((x)
((x)
S(x, 4) = (Sxx (1)))++yy))) 144
(1I)
(xx 7(v)++
where ||lh+ == (Jx (()) न..
The expected pseudo-spherical score under a reference distribution p is defined as:
Epia) [1(x)
Spc == Eppn)S((x (xx f(y)
(12)
Example 2.
The classic spherical scoring rule 19 is special case in the pseudo-spherical family, which corresponds to y
7(x)
7(x)
S(x,
(13)
(xx ( dyy)
|gl|2
The family of pseudo-spherical scoring rules is appealing because it introduces different and principled way for assessing a probability forecast. For example, the spherical scoring rule has an interesting geometric interpretation. Suppose the sample space X' contains mutually exclusive and exhaustive outcomes. Then a probability forecast can be represented as a vector q = (qı,-..,4) ........),))) 4n)- Let vector p (Pı,-.., Pa) represent the oracle probability forecast. The expected spherical score can be written as:
S(p, q) = (Sx, E. Pig:
(p.
|pI|z
= lpllz
(14)
VL.
where (p. q) and L(p, q) denote the inner product and the angle between vectors p and q respectively, In other words, when we want to evaluate the expected spherical score probability forecast under real data distribution p using samples, the angle between p and q is the sufficient statistics.
Since we know that both p and q belong to the probability simplex P = {w) Eaex v(x) = 1 and Va: E X, uxx) 2 0.}. the expected score will minimized if and only the angle zero, which implies p q. More importantly, since all need to do is to minimize the angle of

---

the deviation, we are allowed to scale q by a constant. Specifically, when is an energy baaeyd distribution q = exp(-E.)
E, exp(- - exp(-E3 we can instead evaluate and minimize the angle between data distribution p and the unnormalized distribution q = (cxp(-E1),.... cxpp exp(-En)) Em)). since ((p, q) = L(p, q). More generally, we have the following theorem to justify the use of pseudo-spherical scoring rules for training energy-based models:
exp(-E,, E.,)
Theorem 26 70). Pseudo-spherical scoring rule strictly proper and homogeneous.
As the original definition of pseudo-spherical scoring rule (Equuttion (T)) takes the form of a fraction, for computational considerations, in this paper we instead focus on optimizing its composite scoring rule (Definition 2.1 in 44)):
Definition 4 (7-score (20)). For the expected aseudo-sphericee score Sps(p. q) defined in Equa- tion 12 with y 0, the expected 7-score defined as:
(.(p.g) = log (og((p((P..)) 9)) log (E (Ex())(((z)))) log(Ilal]++1)
(15)
Since log(u) strictly increasing in u, Sy 9) is a strietly proper homogeneous composite score:
max (p. = arg max - log (pp 9)) = arg max (p. = p.
(16)
PEP
PEP 7
PEP
3.2 Pseudo-Spherical Contrastive Divergence
Suppose we parametrize the energy-based model distribution as go x To = exp(-EE) and we want minimize the negative 7-score in Equation (15
1 log (o + log lx+1)
min (, = min 7
(17)
In the following theorem, we derive the gradient of (0; p) with respect to A: Theorem 2. The gradient of (0:pp) with respect to 0 can be written as:
(0:)(0:p) = - Ve log (exp(- ?
Bro( Vo
(18)
where the auxiliary distribution ro is also an energy-based distribution defined as:
[x re(x) Jx (0( dx Jx exp(-(7 1)Eo 1))
1)
In App. B.1 provide two different ways to prove the above theorem. The first one more straightforward and directly differentiates through the term (g([1oollx+1) The second one leverages variational representation of log( [ell where the optimal variational distribution happens to take analytical form of (x)))) (x) thus avoiding the minimax optimization in other variational frameworks such as [89] 12]. The main challenge in maximizing "-score is that is
generally intractable to exactly compute the gradient of the second term in Equation (15).
During training, estimating the second term of Equation (18) requires us to obtain samples from the auxiliary distribution distribution ge X exp(-Ee) that approximates the data distribution. Due to the restrict regularity conditions the convergence of Langevin dynamics, in practice, we found challenging to use the iterative sampling process in Equation 2 with a fixed number of transition steps and step size to produce samples from ro and qo simultaneously, as the temperature 7 + 1 in re simply amounts to a linear rescaling to the energy function during training. Thus for generality, as in contrastive divergence [37 17 [6) and f-EBM (89), make the minimal assumption that we only have sampling procedure to produce samples from Go for both learning and inference procedures.
exp(- 1) while at test time, we want to sample from the model
In this case, we can leverage the analytical form of re and nelf-normelized importance sampling 68 (which been used to derive gradient estimators in other contexts such importance weighted
autoencoder [8) to obtain consistent estimation of Equation (18)
5

---

Algorithm Pseudo-Spherical Contrastive Divergence.
1: Input: Empirical data distribution Pdata- Pseudo-spherical scoring rule hyperparameter
2: Initialize energy function Eg.
3: repeat
4: Draw a minibatch of samples (......, xv) from Pdata-
5: Draw a minibatch samples {æj ,Ny' from go x (xp(-Eg) (e.g., using Langevin dynamics with sample replay buffer).
6: Update the energy function by stochastic gradient descent:
N
Em exp(-> Vo E exp( - (x,
VoLN -o log
Σ (exp((-BB(xt))) 1=1
7: until Convergence
Theorem 3. Let x ..... xy be i.idd samples from p(z) and ......, aN be i.idd. samples from
(0(x) exp(- exp(-Eo(x)). Define the gradient estimator
N
2i=1 wa ) VoEo(
V VoL. (0:p) == - (0:):
log
Σ
exp((>FRo(
(19)
Σi=1 wa W where the self-normalized importance weight we we(x)) F
[2; exp( Eg }).
Then the gradient estimator converges to the true gradient (Equation (181) in probability, i.e. >
lim P VI - (0. 2
=0 0.
N-000
We summarize the pseudo-spherical contrastive divergence (PS-CD) training procedure in Algo- rithm T In Appendix [ we also provide simple Py'Torch implementation for stochastic gradient descent (SGD) with the gradient estimator in Equation (19).
3.3
Connections to Maximum Likelihood Estimation and Extension to y <
From Equation 9 in Section 2.3 we know that y-score induces the following Bregman divergence (the divergence function associated with proper composite scoring rule is analogously defined Def.
2.1 in 44)):
(p qo) (p (p.
and maximizing score is equivalent to minimizing D, (p. qe). In the following lemma, we show that when y + 0, (p, (p. qe) will recover the KL divergence between p and go. and the gradient of PS-CD will recover the gradient of CD.
Lemma 1. When 7 - 0, we have:
lim D. (p. (pl(g))));
lim (o.:(0:)) VOLMLE(O:p). 700
Inspired by [86. [6]] that generalize Rényi divergence beyond its definition to negative orders, we now consider the extension 7-score with ? (although it may not be strictly proper for these ? values). The following lemma shows that maximizing such scoring rule amounts to maximizing lower bound of logarithm score (MLE) with additional Rényi entropy regularization.
Lemma 2. When -1 S < O, we have:
Sy 4) < Fytta flog ((x))] + 7
7 +1y+((1) (g)
where Hy+1(9) the Rényi entropy of order y 1.
Theoretical Analysis
In this section, gain deeper understanding of our PS-CD algorithm and how the proposed estimator behaves, we analyze its sample complexity and convergence property under certain conditions. All the proofs for this section can be found in Appendix

---

4.1 Sample Complexity
We start with analyzing the sample complexity the consistent gradient estimator in Equation 19 that is how fast it approaches the true gradient value. We first make the following assumption: Assumption 1. The energy function is bounded by K and the gradient is bounded by L:
Vx E X, 0 θ [Eo()) - K, [V L.
This assumption is typically easy satisfy because in practice we always use a bounded sample space (e.g. normalizing images to [0, 1] or truncated Gaussian) to ensure stability. For example, in image modeling experiments, we use L2 regularization on the outputs of the energy function (hence bounded energy values), as well as normalized inputs and spectral normalization [60] for the neural network that realizes the energy function (hence Lipschitz continuous with bounded gradient).
With vector Bernstein inequality 47
32
we have the following theorem showing a sample complex-
ity of
1oR(1/6) such that the estimation error is less than with probability at least
Theorem 4. For any constants e > 0 and (0, 1), when the number of samples N satisfies:
32 (1 + 4 4log(2/6)
2²
we have:
( VOL. (o:)(0:))
sc) 21-6
4.2 Convergence
Typically, convergence of SGD are analyzed for unbiased gradient estimators, while the gradient estimator in PS-CD is asymptotically consistent but biased. Building on the sample complexity bound above and prior theoretical works for analyzing SGD (8 24], we analyze the convergence of PS-CD. For notational convenience, we use [(0) to denote the loss function L. S.,((p go). Besides Assumption
[ Assumption 2. The loss function C(0) is M-smooth (with M 0):
we further make the following assumption on the smoothness of L(():
401, 02 O [VV(())) (L s M| 0211
This is common assumption for analyzing first-order optimization methods, which is also used in Also this is a relatively mild assumption since we do not require the loss function to be convex 0. Since in non-convex optimization, the convergence criterion is typically measured by gradient norm, following [64] 24), we use || C())l to judge whether a solution O is approximately stationary point.
24 S
(0,1) and 8 (0, suppose that the step sizes satisfy < 2(1 - -a)M and the sample size N, used for estimating 9t is sufficiently large (satisfying Equation (36)). Let L denote the minimum value of C(0). Then with probability at least the
Theorem 5. For any constants
output of Algorithm
(in Appendix C.2). 0, satisfies (constant C == e
- +2C Σ 1
E V
Σt. (2(1 a)) M
The above theorem implies the following corollary, which shows a typical convergence rate of O(1/VT) for non-convex optimization problems:
Corollary 1. Under the conditions in Theorem exceept that we use constant step sizes: т)
min((1 - a//M,1/VT) for 1,...., Then with probability at least 1 5, we have (constant
C:= aM
2M((())) L2) -())
-() 12C
(1-a)21
(1 a)aa
In Appendix C.2 we discuss more on the strongly convex (Theorem) 7 and convex cases (Theorem
7

---

5 Related Work
Direct KL Minimization. Under the "analysis by synthesis" scheme, [37 proposed Contrastive
Divergence (CD), which estimates the gradient of the log-partition function (arising from KL) using samples from some MCMC procedure. To improve the mixing time of MCMC. [17 proposed to employ Persistent CD and a replay buffer to store intermediate samples from Markov chains throughout training, and [66] proposed learn non-convergent short-run MCMC. Both approaches (long-run and short-run MCMC) work well with PS-CD in our experiments. PS-CD may also benefit from recent advances on unbiased MCMC 44 73
which we leave as interesting future work.
Fenchel Duality. By exploiting the primal-dual view of KL, recent works cast maximum likelihood training of EBMs as minimax problems, which introduce a dual sampler and are approximately solved by alternating gradient descent ascent updates. Along this line, to allow flexible modeling preferences, (89 proposed f-EBM enable the use of any f-divergence to train EBMs, which also relies on minimax optimization By contrast, in this work, we leverage the analytical form of the optimal variational distribution and self- nell-normllezed importance sampling to reach a framework that requires no adversarial training and has no additional computational cost compared CD while allowing flexible modeling preferences. Besides convenient optimization, PS-CD and f-EBM trains EBMs with two different families of divergences (hence complementary) with KL being the only shared one, since any pseudo-spherical scoring rule corresponds to Bregman divergence (Section 2.3) and the only member in f f-divergence that is also Bregman divergence is a-divergence (with KL as special case) (Theorem 4 in [I).
proposed to
Homogeneous Scoring Rules. 84 proposed to learn unnormalized statistical models on discrete sample space by maximizing 7-score, which uses empirical data distribution (f(z) R2 where ne is the number of appearance of æ in the dataset and n is the total number of data) as a surrogate to the real data density p and relies on a localization trick to bypass the computation of || Il Consequently, it is only amenable to finite discrete sample space such as natural language [51]. whereas PS-CD is applicable
any unnormalized probabilistic model in continuous domains. Another popular homogeneous scoring rule is the Hyvärinen score, which gives rise to the score matching objective (39 for EBM training. However, score matching and its variants [8 have difficulties in low data density regions and do not perform well in practice when training EBMs on high-dimensional datasets [80]. Moreover, since the score matching objective involves the Hessian of log-density functions that is generally expensive to compute (58), methods such as approximate propagation [45), curvature propagation (58) and sliced score matching [83 are needed to approximately compute the trace of the Hessian.
Noise Contrastive Estimation. Another principle for learning EBMs is Noise Contrastive Esti- mation (NCE) [A), where an EBM is learned by contrasting prescribed noise distribution with tractable density against the unknown data distribution. Using various Bregman divergences, NCE can be generalized to a family of different loss functionals [33 85]. However, finding an appropriate noise distribution for NCE is highly non-trivial. In practice, NCE typically works well conjunction with carefully-designed noise distribution such as context-dependent noise distribution 41 or joint
learning of a flow-based noise distribution (1
In this work, we focus on generalizing maximum likelihood by deriving novel training objectives for EBMs without involving auxiliary models (e.g., the variational function in [89] the flow-based noise
distribution in 21 and the amortized sampler in [50 12 29
Experiments
In this section, we demonstrate the effectiveness of PS-CD on several 1-D and 2-D synthetic datasets as well as commonly used image datasets.
Setup. The 2-D synthetic datasets include Cosine, Swiss Roll, Moon, Mixture of Gaussian, Funnel
and Rings, which cover different modalities and geometries (see Figure 2 in App. D.I for illustration).
To test the practical usefulness, we use MNIST [54), CIFAR-10 48) and CelebA [57] in our
experiments for modeling natural images. Following [0], for CelebA, we first center-crop the images
to 140 x 140, then resize them to 64 x 64. More experimental details about the data processing,
model architectures, sampling strategies and additional experimental results can be found in App. D
8

---

Effects of Different Values. To illustrate the model- ing flexibility brought by PS-CD and provide insights on the effects of different values, we first conduct 1-D synthetic experiment similar to the one in [89]. As shown in Figure [T, when fitting mixture of Gaussian with a single Gaussian (i.e., model mis-specification case), the family of PS-CD offers flexible tradeoff's between quality and diversity (i.e., mode collapse vs. mode coverage). Although in the well-specified case
paget of
Figure 1: The effects of different 7 values when fitting a mixture of Gaussian with a single Gaussian.
these objectives induce the same optimal solution, in
this example, we can see that a larger y leads to higher entropy. More importantly, compared to f-EBM (89
that also provides similar modeling flexibility and includes CD special case, PS-CI> does not require expensive and unstable minimax optimization (no additional computational cost compared to CD). In App. D.2] further visualize the objective landscapes for different " values. As shown in Figure 3 and 4 when the model is well-specified, different objectives will induce the same optimal solution since they are strictly proper; when the model is mis-specified (corresponding to practical scenarios), different objectives will exhibit different modeling preferences.
Furthermore, better demonstrate the modeling flexibility brought by PS-CD in high- sigh-dimessionl case, we conduct experiments on CIFAR- 10 using a set of indicative and reliable metrics (Density, Coverage, Precision, Recall) proposed by [6] to evaluate the effects of from various perspectives. Please refer to App. D.3 for experimental results (Table 4 and detailed discussions.
2-D Synthetic Data. For quantitative evaluation of the 2-D synthetic data experiments, we follow [79] and report the maximum mean discrepancy (MMD, [5) be- tween the generated samples and validation samples in Table in App. D.I which demonstrates that PS-CD outperforms its CD counterpart on all but the Funnel dataset. From the histograms of samples shown in we can also have similar obser- vations. For example, CD fails to place high densities in the center of the right mode in MoG, while PS-CD places the modes correctly.
Table 1: FID scores for CIFAR-10 10 condi- tional and CelebA unconditional image gen- eration. We list comparisons with results reported by CD 177. Noise-Condition Score Network (NCSN) [8]] and f-EBMs 88]. y = 1.0 corresponds to maximizing spherical scoring rule (Example 2
Figure 2 in App.
Method
FID
CIEAR-10 (32 x 32) Conditional
Image Generation. In FigureB]jn A.p we show MNIST, CIFAR-10 and CelebA samples produced by
Contrastive Divergence (KL)
37.90
f-EBM (KI.)
37.36
PS-CD (with 7 = 1.0), which demonstrate that our approach can produce highly realistic images with sim- ple model architectures. As suggested in [17), we use Fréchet Inception Distance (FID) 36) as the quantita- tive evaluation metric for CIFAR-10 10 and CelebA, as Langevin dynamics converge to local minima that ar- tificially inflate Inception Score 777). From Table we can see that various members (different 7 values) in the family of PS-CD can outperform CD significantly. and more surprisingly, PS-CD also shows competitive performance to the recently proposed f-EBMs, with- out requiring expensive minimax optimization. While our method currently does not outperform the state- of-the-art image generation methods such as improved denoising score matching [8I], which relies on care- fully selected noise schedule and specially designed noise-dependent score network (modified U-Net architecture, hence not directly comparable to our
f-EB (Reverse KL)
33.25
f-BBM (Squared Hellinger)
32.19
f-BBM (Jensen Shannon)
30.86
Pseudo-Spherical CD (7 2.0)
33.19
Pseudo Sruudo-Speeriaal CD (7 1.0)
29.78
Pscudo- Sauud--ppeerical CD (7 = 0.5)
35.02
Pseudo-Spherical CD (7 -0.5)
27.95
CelebA (64 x 64)
Contrastive Divergence (KL)
26.10
NCSN (w/o denoising)
26.89
NCSN (w/ denoising)
25.30
NCSNv2 (w/o denoising)
28.86
NCSNv2 (w/ denoising)
10.23
Pseudo-Spherical CD (y = 1.0)
24.76
Sphuri-Spherical CD (7- = 00.5)
20.35
results), we think that our work opens up a new research direction by bridging statistical decision theory (homogeneous proper scoring rules) and deep energy-based generative modeling. Moreover,
For CelebA dataset, we reproduced the short-run MCMC method (66 using our code base. Moreover, the f-BBM paper only reported results on CelebA 32x32 and we empirically found it is not comparable to our method in CelebA 64x64 (higher resolution). indicating better scalability of PS- PD-CD to high-dimensional case.

---

Table 2: Robustness to data contamination on Gaussian datasets. The data distribution is (V, 0.5) and the contamination distribution is (2, 0.05). We measure the KL divergence between clean target distribution p and converged model distribution go, DKI. (pllgo)
Contamination Ratio
CD
PS-CD (7 = 0.5)
PS-CD (7 = 1.0)
PS-CD ( = 2.0)
0.01
0.0067
le-5
le-7
le-6
0.05
0.0851
0.00027
1.6e-6
0.0001
0.1
0.1979
0.00173
1.86e-6
0.00012
0.2
0,3869
0.1858
6.4e-6
0.00017
0.3
0.5438
0.5429
0.3118
0.00029
under the setting of simple model architectures and the same hyperparameter configuration (e.g.. batch size, learning rate, network structure, etc.), our empirical results suggest clear superiority of PS-CD over traditional CD and recent f-EBMs.
OOD Detection & Robustness to Data Contamination. We further test our methods on out-of- distribution (OOD) detection tasks. For the conditional CIFAR-I0 model, follow the evaluation protocol in [17 and use s(x) = m - E' y) as the score for detecting outliers. We use SVHN 663. Textures D Uniform/Gaussian Noise, C-FAR--0 Linear Interpolation and CelebA as the OOD datasets. We summarize the results in Table 5 in App. D.6 from which we can see that PS-CD
consistently outperforms CD and other fikelihood-basee models.
Inspired by the OOD detection performance and previous work on robust parameter estimation under data contamination (43), we further test the robustness of CD and PS-CD both synthetic and natural image datasets. Specifically, suppose p(æ) is the underlying data distribution and there is another contamination distribution w (z)), e.g. uniform noise. In generative modeling under data contamination, our model observes i.i.d. samples from the contaminated distribution p(x:) = cp(x:) + (1 (x), where 1 c E [0, 1/2) is the contamination ratio. A theoretical advantage of pseudo-spherical score is its robustness to data contamination: the optimal solution of ming Sps (p. qo) is close to that of ming (p. 9a) under some conditions (e.g. the density of w(x) mostly lies in the region for which the target density p(z) is small) (20 43). From Table 2 we can see that CD suffers from data contamination severely: as the contamination ratio increases. the performance degrades drastically. By contrast, PS-CD shows good robustness against data contamination and a larger y leads to better robustness. For example, PS-CD with y = 1.0 can properly approximate the target distribution when the contamination ratio is 0.2, while PS-CD with y 2.0 can do so when the contamination ratio is up to 0.3.
We conduct similar experiments on MNIST and CIFAR-10 datasets, where we use uniform noise as the contamination distribution and the contamination ratio is 0.1 (i.e. 10% images in the training set are replaced with random noise). After warm-up pretraining 77 (when the model has some OOD detection ability). we train the model with the contaminated data and measure the training progress. We observe that CD gradually generates more random noise and diverge after few training steps, while PS-CD is very robust. As shown in Table in App. D.6 for a slightly pre-trained unconditional CIFAR-10 model (a simple 5-layer CNN with FID of 68. 77), we observe that the performance of CD degrades drastically in terms of FID, while PS-CD can continuously improve the model even using the contaminated data. We provide visualizations and theoretical explanations in App. D.6
7 Conclusion
From the perspective of maximizing strictly proper homogeneous scoring rules, we propose pseudo- spherical contrastive divergence (PS-CD) to generalize maximum likelihood estimation of energy- based models. Different from prior works that involve joint training of auxiliary models [89] 21 [50]
12 29,, PS-CD allows us to specify flexible modeling preferences without additional computational
cost compared to contrastive divergence. We provide a theoretical analysis the sample complexity and convergence property of the proposed method, as well as its connection to maximum likelihood. Finally, we demonstrate the effectiveness of PS-CD with extensive experiments both synthetic data and commonly used image datasets.
Note that it is impossible for a randomly initialized model to be robust to data contamination since without additional inductive bias, it will simply treat the contaminated distribution p(æ)) the target.
10

---

Acknowledgements
This research was supported by NSF(#1651565, #1522054, #1733686), ONR (N000141912145), AFOSR (FA95501910024), ARO (W911NF-21-1-0125) and Sloan Fellowship.
References
[1] Shun-ichi Amari. Divergence functiio, information monotonicity and information geometry. In Workshop on information theoretic methods in science and engineering (WITMSE). Citeseer, 2009.
[2] Michael Arbel, Liang Zhou, and Arthur Gretton. Kale: When energy-based learning meets adversarial training. arXiv preprint arXiv:2003.05033, 2020.
[3] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan. arXiv preprint
a71107701..07875 2017.
[4] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. Mine: mutual information neural estimation. arXiv preprint arXiv: 10fiv.8800.00m 04062, 2018.
[5] Karsten Borgwardt, Arthur Gretton, Malte Rasch, Hans-Peter Kriegel, Bernhard Schölkopf, and Alex J Smola. Integrating structured biological data by kernel maximum mean discrepancy. Bioinformatics, 22(14):c49-e577 2006.
[6] Lev M Bregman. The relaxation method of finding the common point of convex sets and its application the solution of problems in convex programming. USSR computational mathematics and mathematical physics, 7(3):200-217, 1967.
[7] Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. arXiv preprint arXiv: 550.00059, 2015.
[8] Jie Chen and Ronny Luss. Stochastic gradient descent with biased but consistent gradient 2018.
estimators. arXiv preprint
[9] Mircea Cimpoi, Subhransu Maji, Jasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3606-3613, 2014.
[10] Imre Csiszár, Eine informationsthoorrtichee ungleichung und ihre anwendung auf beweis der ergodizitaet von markoffschen ketten. Magyer Tud. Akad, Mat. Kulato Int. Koezl., 8:85-108, 1964.
[11] Bo Dai. Hanjun Dai, Arthur Gretton. Le Song, Dale Schuurmans, and Niao He. Kernel exponential family estimation via doubly dual embedding, arXiv preprint arXiv:1811.02228. 2018.
[12] Bo Dai, Zhen Liu, Hanjun Dai, Niao He, Arthur Gretton, Le Song, and Dale Schuurmans. Expo- nential family estimation via adversarial dynamics embedding. arXiv preprint arXiv: 190..1204........ 2019.
[13] A Philip Dawid. Coherent measures of discrepancy, uncertainty and dependence, with applica- tions to bayesian predictive experimental design. Department of Statistical Science, University College London. htlp://www: ucl. ac. n//Suat//rseuurccdaa5544 himl, Tech. Rep, 139, 1998.
[14] Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516, 2014.
[15] Laurent Dinh, Jascha
and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv: 1605 08803, 2016.
[16] Yilun Du, Toru Lin, and Igor Mordatch. Model based planning with energy based models. arXiv preprint 1909. 2019.

---

[17] Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models, In Advances in Neural Information Processing Systems 32, pages 3603-3613, 2019.
[18] Axel Finke and Alexandre H Thiery. On importance-weighted autoencoders. arXiv preprint arXiv: 10477, 2019.
[19] Daniel Friedman. Effective scoring rules for probabilistic forecasts. Management Science, 29(4)4447-454, 1983.
[20] Hironori Fujisawa and Shinto Eguchi. Robust parameter estimation with small bias against heavy contamination. Journal of Multivariate Analysis, 99(9):2053 2008.
[21] Ruiqi Gao, Erik Nijkamp, Diederik P Kingma, Zhen Xu, Andrew M Dai, and Ying Nian Wu. Flow contrastive estimation of energy-based models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7518-7528, 2020.
[22] Paul H Garthwaite, Joseph B Kadane, and Anthony O' Hagan. Statistical methods for eliciting probability distributions. Journal of the American Statistical Association, 100(470):680-011,
2005.
[23] Mathieu Germain, Karol Gregor, lain Murray, and Hugo I Larochelle. Made: Masked autoencoder for distribution estimation. In International Conference on Machine Learning, pages 881-889, 2015.
[24] Saced Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23(4):3341 2013.
[25] Alison Gibbs and Francis Edward Su. On choosing and bounding probability metrics. International statistical review, 70(3):419-435 2002.
[26] Tilmann Gneiting and Adrian E Raftery. Strietly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359-3777 2007.
[27] Good. Comment "measuring information and uncertainty" by robert buehler. Foundations of Statistical Inference, pages 337-339, 1971.
[28] Ian Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets, In Advances in neural information processing systems, pages 2672-2680, 2014.
[29] Will Grathwohl, Jacob Kelly, Milad Hashemi, Mohammad Norouzi, Kevin Swersky, and David Duvenaud. No meme for me: Amortized sampling for fast and stable training of energy-based based models. arXiv preprint arXiv:2010.044: 044230, 2020.
[30] Will Grathwohl, Kuan-Chieh Wang, Vörn-Henrit Jacohsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. arXiv preprint arXiv: /9/2.03///000666 2019.
[31] Ulf Grenander, Michael Miller, Michael Miller, et al. Pattern theory: from representation to inference. Oxford university press, 2007.
[32] David Gross. Recovering low-rank matrices from few coefficients in any basis. IEEE Transac- lions on Information Theory, 57(3):1548-1566 2011.
[33] Michael U Gutmann and Jun-ichiro Hirayama. Bregman divergence as general framework to estimate unnormalized statistical models. Proceedings of the TiwentySeventhh Conference on Uncertainty in Artificial Intelligence, pages 283-290, 2011.
[34] Michael U Gutmann and Aapo Hyvärinen. Noise-contrastive estimation of unnormalized statistical models, with applications natural image statisties. The journal of machine learning research, 13(1):307-361, 2012.
[35] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
12

---

[36] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler. and Sepp Hochreiter. Gans trained by a two time-scall scale update rule converge to a local nash equilibrium. In Advances in neural information processing systems, pages 6626-6637, 2017.
[37] Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771-1800, 2002.
[38] Tito Homem-de Mello. On rates of convergence for stochastic optimization problems under non- independent and identically distributed sampling, SIAM Journal on Optimization, 19(2):524- 551, 2008
[39] Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(Apr):695-70) 2005.
[40] Pierre E Jacob, John O'Leary, and Yves F Atchadé. Unbiased markov chain monte carlo with couplings. arXiv preprini urXiv: / 036.03625, 2017.
[41] Shihao Ji, SVN Vishwanathan, Nadathur Satish, Michael Anderson, and Pradeep Dubey. Blackout: Speeding up recurrent neural network language models with very large vocabularies. 2015.
arXiv preprint arXiv:
[42] Chi Jin, Praneeth Netrapalli, and Michael Jordan. What is local optimality in nonconvex- nonconcave minimax optimization? arXiv preprint arXiv: wwxiv:/92..00//8 2019.
[43] Takafumi Kanamori and Hironori Fujisawa. Robust estimation under heavy contamination using unnormalized models. Biometrika, 102(3):559-577 2015.
[44] Takafumi Kanamori, Hironori Fujisawa, et al. Affine invariant divergences associated with proper composite scoring rules and their applications. Bernoulii, 20(4):2278-22) 22002244,, 2014.
[45] Durk P Kingma and Yann L. Cun. Regularized estimation of image statistics by score matching. In Advances neural information processing systems, pages 1126-1134, 2010.
[46] Durk Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions.
In Advances in Neural Information Processing Systems, pages 10215-10224, 2018.
[47] Jonas Moritz Kohler and Aurelien Lucchi. Sub-sampled cubic regularization for non-convex optimization. In Proceedings of the 34th International Conference on Machine Learning-Voii Volume 70, pages 1895-1904. JMLR. org, 2017.
[48] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[49] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097-1105, 2012.
[50] Rithesh Kumar, Sherjil Ozair, Anirudh Goyal, Aaron Courville, and Yoshua Bengio. Maximum entropy generators for energy-based models. arXiv preprint arXiv:/901 0085008 2019.
[51] Matthieu Labeau and Shay B Cohen. Experimenting with power divergences for language modeling. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4095-4105, 2019.
[52] Simon Julien, Mark Schmidt, and Francis Bach. A simpler approach to obtaining an (1/t) convergence rate for the projected stochastic subgradient method. arXiv preprint ArXiv: 2012.
[53] Hugo Larochelle and lain Murray. The neural autoregressive distribution estimator. In Proceed- ings of the Fourteenth International Conference Artificial Intelligence and Statistics, pages 29-37, 2011.
[54] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE,
1998.
13

---

[55] Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
[56] Yingzhen Li and Richard E Turner. Rényi divergence variational inference. In Advances in Neural Information Processing Systems, pages 1073-1081, 2016.
[57] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[58] James Martens, Ilya Sutskever, and Kevin Swersky. Estimating the hessian by back poocppropggtting curvature. arXiv preprint arXiv: 2206.4464, 2012.
[59] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? arXiv preprint arXiv: /tfl:/800..000 0044666 2018.
[60] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks, arXiv preprint arXiv:1802.055: 2018.
[61] Muhammad Ferjad Nacem, Scong Joon Oh, Youngjung Uh. Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. In International Conference on Machine Learning, pages 7176-7185. PMLR, 2020.
[62] Radford M Neal et al. Memc using hamiltonian dynamics. Handbook of markov chain monte carlo, 2(11):2, 2011.
[63] Arkadi Nemirovski, Anatoli Juditsky, Guanghui Lan, and Alexander Shapiro. Robust stochastic approximation approach to stochastic programming, SIAM Journal on optimization, 19(4):1574- 1609, 2009.
[64] Yurii Nesterov. Introductory lectures on convex optimization: A basic course, volume 87. Springer Science & Business Media, 2013.
[65] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[66] Erik Nijkamp, Mitch Hill, Song- Chun Zhu, and Ying Nian Wu. Learning non-convergent non-persise non-persistent short-run meme toward energy-based model. In Advances in Neural Information Processing Systems, pages 5233-5243, 2019.
[67] Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural sam- plers using variational divergence minimization. In Advances in neural information processing systems, pages 271-279, 2016.
[68] Art B. Owen. Monte Carlo theory, methods and examples. 2013.
[69] Matthew Parry, A Philip Dawid, Steffen Lauritzen, et al. Proper local scoring rules. The Annals of Statisties, 40(1):561-592, 2012.
[70] Matthew Parry et al. Linear scoring rules for probabilistic binary classification. Electronic Journal of Statistics, 10(1):199--1077 2016.
[71] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, -egghperformncce deep learning library. In Advances in Neural Information Processing Systems, pages 0024-8035. 2019.
[72] Hoifung Poon and Pedro Domingos. Sum-product networks: A new deep architecture. In 201 I IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pages 689-690. IEFE, 2011.
[73] Yixuan Qiu, Lingsong Zhang, and Xiao Wang. Unbiased contrastive divergence algorithm for training energy-based latent variable models.
14

---

[74] Sashank J Reddi, Ahmed Hefny, Suvrit Sra, Barnabás Póczos, and Alex Smola. Stochastic variance reduction for nonconvex optimization. International conference on machine learning. pages 314-323, 2016.
[75] Christian Robert and George Casella. Monte Carlo statistical methods. Springer Science & Business Media, 2013.
[76] Thornton B Roby. Belief states and the uses of evidence. Behavioral Science, 10(3))255-270, 1965.
[77] Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pages 2234-2242, 2016.
[78] Shai Shalat-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
[79] Jiaming Song and Stefano Ermon. Bridging the gap between f-gans and wasserstein gans. arXiv preprint arXiv: 2019.
[80] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, pages 11895-11907, 2019.
[81] Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. In Advances in Neural Information Processing Systems, 2020.
(82) Yang Song, Sahaj Garg. Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Proceedings of the Thirty-Fifhh Conference on Uncertainty in Artificial Intelligence, UAI 2019, Tel Aviv, Israel, July 22-25, 2019, page 204, 2019.
[83] Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Uncertainty in Artificial Intelligence, pages 574-584. PMLR, 2020.
[84] Takashi Takenouchi and Takafumi Kanamori. Empirical localization of homogeneous diver- gences on discrete sample spaces. In Advances in Neural Information Processing Systems, pages 820-828, 2015.
[85] Masatoshi
Uehara, Takafumi Kanamori, Takashi Takenouchi, and Takeru Matsuda. A uni- fied statistically efficient estimation framework for unnormalized models. In International Conference Artificial Intelligence and Statistics, pages 809-819, 2020.
[86] Tim Van Erven and Peter Harremos. Rényi divergence and kullback-leibler divergence. IEEE Transactions on Information Theory, 60(77)3777-338) 372-322, 2014.
[87] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, (37):1661111674 2011.
[88] Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 681-688, 2011.
[89] Lantao Yu, Yang Song, Jiaming Song, and Stefano Ermon. Training deep energy-based models with f-divergence minimization. In International Conference on Machine Learning, pages 10957-10967. PMLR, 2020.
[90] Jun Zhang, Yaokun Lei, Yi Isaac Yang, and Yi Qin Gao. Deep learning for multi-scale molecular modeling. 2020.
[91] Shengjia Zhao, Hongyu Ren, Arianna Yuan, Jiaming Song, Noah Goodman, and Stefano Ermon. Bias and generalization in deep generative models: An empirical study. In Advances in Neural Information Processing Systems, pages 10792-10801, 2018.
15


Reviews:
[{"review_id": "kQob708U08g", "reviewer": "Reviewer_CEzj", "summary": "This paper provides an efficient framework to estimate energy based model (EBM). \nWhile conventional KL or general f-divergence-based contrastive divergence learning suffers from instability and non-convergence issues, the proposed framework namely pseudo-spherical contrastive divergence (PS-CD) can be implemented as SGD type algorithm whose sample complexity and convergence property are provided. \nPS-CD includes the conventional CD as a special case and thus provides gradient-based optimization framework for CD. \nExperiments to evaluate the effectiveness of the proposed approach are conducted with both synthetic and image benchmark datasets. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The proposed method introduces the PS score in statistical decision theory for learning EBM. The proposed method, PS-CD, is defined as 1/\\gamma times the negative logarithmic PS scoring rule for parameter \\gamma. The advantage of PS-CD is that a consistent estimator of its gradient can be easily computed, allowing EBMs to be trained by gradient methods. This is an advantage of using the PS scoring rule, an approach that has not been seen in previous learning methods. \nSince PS-CD reproduces ordinary CD by adjusting the parameter \\gamma, it can be interpreted that the proposed method is a generalization of CD and also provides learning by gradient method.\n\nQuality: This paper technically sounds. Although the main contribution of the paper is the proposal of a learning framework and the investigation of its theoretical properties, minimal experimental results and discussion are also presented to evaluate the effectiveness proposed method.\n\nClarity: This paper is well written. It clearly explains why a more general framework is needed (what are issues with conventional methods), the process of deriving the proposed method, and its theoretical considerations.\n\nSignificance:For the learning of EBMs discussed in this paper, the CD method, which is the maximum likelihood learning based on KL divergence, and the f-EBM, which minimizes the more general f-divergence, have been usually used. These methods had difficulties such as unstable learning and complex variational computation. The presented work that avoids these issues and enable learning by the gradient method is considered to have a high impact (\nthe derived sample complexity and convergence rate show that the proposed method can theoretically achieve the similar performance as that of learning by ordinary SGD). \n\nQuestions:\n- In the experiment, the authors discuss the effect of the parameter \\gamma in the case of model misspecification. How much of the effect of the model-misspecification can be absorbed by adjusting the \\gamma? From the experimental results, I think that the adjustment of \\ gamma does not contribute much to it.\n\n- While negative \\gamma values seem to be important from the experimental results, Section 3.3 points out that they may not be strictly proper (scoring rule). If so, how does a negative \\gamma affect theoretical results derived in this paper?\n\n- I think it would be better to explain the key idea of convergence analysis based on biased gradient estimator in the main text.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "While it provides a general framework for learning EBMs, it is not as good as SOTA methods in terms of FID score (although I believe that it is not appropriate to evaluate generative models with a single score).\nI believe that there is no negative social impact from this research.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "20h"}, "extra_text": {}}, {"review_id": "R9tHZwx4N-J", "reviewer": "Reviewer_y75x", "summary": "The authors propose a spectrum of objectives for learning energy-based models that generalise maximum likelihood. These objectives--known as pseudo-spherical scoring rules--are similar to (persistent) contrastive divergence in terms of their implementation and computational cost, but have an extra degree of flexibility that can boost downstream performance for e.g. image generation.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Overall verdict**\n\nThis paper contains some solid results, with no obvious technical shortcomings. Personally, I didn't find the method especially exciting and the experiments were slightly limited, leaving me with concerns about the significance/impact of the paper. Nevertheless, I lean towards acceptance.\n\n**Originality + Quality + Significance**\n\nThis proposed method, PS-CD, is similar in spirit to f-EBM [1] insofar as it offers a spectrum of losses that are suitable for training energy-based models. The key differences to f-EBM are:\n \n1) The losses are not based on f-divergences, but rather spherical scoring rules, which correspond to Bregman divergences.\n2) Unlike f-EBM, PS-CD does not require us to solve a minimax problem, and therefore has the same computational cost as persistent contrastive divergence.\n\nI think these contributions are solid, and the accompanying theory & experiments are sufficient to convince me that the method is correct and potentially useful.\n\nI don't think this paper constitutes any kind of fundamental advance for energy-based modelling, but it provides an extra 'knob to twist' when learning EBMs, which *may* give us more control over the kinds of statistical patterns extracted by the EBM. This final claim is supported by the experiments, where we see modest improvements in FID scores compared to f-EBM.\n\nA couple of issues/questions (in order of importance):\n\n1) I found the experiments to be quite limited (2d synthetic density estimation +  image generation on 3 datasets). I would have been much more convinced if you had considered other applications e.g. out-of-distribution detection or joint energy-based modelling and classification as in [5].\n\n2) In line 81, you cite [2] as a 'recent advance...to obtain samples from an EBM'. This is misleading, since the point of that paper was that short-run \"samples\" are not actually valid samples from the EBM (although they can still be high quality). I think this point needs emphasising in the paper, and you should make it clear when reporting results whether or not the samples are likely to be valid. I also think its good practice to report the quality of samples that are obtained after running the MCMC sampler for a long time (e.g. tens of thousands of time steps).\n\n3) I think you should clarify in the main text exactly how your convergence analysis builds on [3] & [4], which you describe as 'prior theoretical works for analysing SGD'. Is it fair to describe theorem 5 as essentially an application of the existing theory? Or are there techniques used in your proofs that are sufficiently non-obvious that readers will learn something by reading them?\n\n4) The theoretical results rely on the sample size growing exponentially with K (which bounds the energy). I imagine that actually using this number of samples is not possible in practice since K would be too large. Do the authors agree? What range of energy values are observed in the experiments? If I am right, then this means the theoretical guarantees have quite limited practical relevance.\n\n5) Equation 19 relies on self-normalised importance sampling. Have you tried measuring the variance / effective-sample-size of this estimator?\n\n**Clarity + prior work**\n\nI think the paper is very well structured & written, and that the references to prior work are extremely thorough. There has been a lot of literature published on EBMs in the last couple of years, and I think the authors cite most of the key papers.\n\n** Minor points **\n\n- Personally, I don't think Theorems 2 & 3 warrant the title 'Theorem'. Just stating the equations would be sufficient for me.\n\n** References **\n\n[1] Lantao Yu, Yang Song, Jiaming Song, and Stefano Ermon. Training deep energy-based models with f-divergence minimization.  InInternational Conference on Machine Learning, pages10957–10967. PMLR, 2020.\n\n[2] Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu.  Learning non-convergent non-persistent short-run mcmc toward energy-based model. InAdvances in Neural Information Processing Systems, pages 5233–5243, 2019\n\n[3] Jie Chen and Ronny Luss.  Stochastic gradient descent with biased but consistent gradient estimators.arXiv preprint arXiv:1807.11880, 2018\n\n[4] Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming.SIAM Journal on Optimization, 23(4):2341–2368, 2013.\n\n[5] Grathwohl, Will, et al. \"Your classifier is secretly an energy based model and you should treat it like one.\" arXiv preprint arXiv:1912.03263 (2019).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I am happy with the author's discussion of limitations and societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "LhFsqnBgSiw", "reviewer": "Reviewer_4NoL", "summary": "The paper introduces a class of divergence to learn EBMs. The class of divergence includes KL as a special case. The paper approximates the gradient of the class of divergence by self-normalized importance sampling and analyzes the convergence when optimizing with the approximate gradient. Experiments are conducted to evaluate the influence of $\\gamma$.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\nThe idea is relatively novel, which extends CD to PS-CD. Under some $\\gamma$ PS-CD outperforms CD. The paper is clearly written.\n\nQuestions:\n1. Do you have a sense how $\\gamma$ influences the FID results in Table 1? How to choose a best $\\gamma$ in practice?\n\n2. In practice, the training of CD is unstable, e.g., the training is likely to diverge (see [a], Appendix H.3). Will the PS-CD overcome these issues?\n\n\n[a] YOUR CLASSIFIER IS SECRETLY AN ENERGY BASED MODEL AND YOU SHOULD TREAT IT LIKE ONE", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "8YUNLwpZAz", "reviewer": "Reviewer_mWGE", "summary": "This paper presents a generalization of the Contrastive Divergence that uses pseudo-spherical scoring rules to define a new family of loss functions for generative modeling. This family of objective functions has an auxiliary loss parameter that can balance the trade-off between diversity and quality. The method is applied to learn EBMs on image benchmark datasets.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main contribution of this paper is to define a new framework for EBM learning by situating the classical CD objective function within a continuous spectrum of loss functions. While a similar concept has been used to generalize EBM learning using $f$-divergences, the new framework presented in the present work is distinct from prior formulations. Furthermore, the PS-CD method proposed in this work encompasses CD as a special case for a certain hyperparameter setting. The theoretical derivation of a novel EBM loss is quite original and the strongest aspect of the paper. \n\nThe proposed method is evaluated on benchmark datasets commonly used in generative modeling. The authors show that adjusting the value of $\\gamma$ to be non-zero tends to give significantly better results than standard CD learning corresponding to $\\gamma = 0$. The authors observe good results for both positive and negative non-zero values of $\\gamma$, and I am unsure of the explanation behind this. I am not sure that conditional Cifar-10 is the best benchmark because it is much less common than unconditional Cifar-10. Unconditional Cifar-10 might be more appropriate since there are a wider variety of existing benchmarks for comparison. In the experiments presented, the PS-CD learning method appears to yield some moderate improvement in the quality of generated images. I am curious if the authors have techniques for tuning $\\gamma$ beyond trying out a few different options.\n\n**Overall**: The proposed PS-CD learning method is original and interesting. Although the empirical results might not show dramatic practical benefits of the learned model, the work is nonetheless a novel contribution and the generalizations of CD used in this work might influence future practitioners.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and societal impact of their work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduced a class of divergence to learn energy-based models. The proposed divergence class has an auxiliary loss parameter that can balance the trade-off between diversity and quality. The method was applied to image benchmark datasets. The reviewers agree that this is an interesting paper with nice contributions. We conclude that the paper is worthy of acceptance. 
Decision: Accept (Poster)