Dual Training of Energy-Based Models with Overparametrized Shallow Neural Networks
Authors: Carles Domingo-Enrich
Alberto Bietti
Marylou Gabrié
Joan Bruna
Eric Vanden-Eijnden
Venue: ICLR 2022 Submitted (2022)
Keywords: energy-based models
generative modeling
neural networks
duality
Fenchel
maximum mean discrepancy
maximum likelihood
active
lazy
score matching
measure
feature

Abstract: Energy-based models (EBMs) are generative models that are usually trained via maximum likelihood estimation. This approach becomes challenging in generic situations where the trained energy is nonconvex, due to the need to sample the Gibbs distribution associated with this energy. Using general Fenchel duality results, we derive variational principles dual to maximum likelihood EBMs with shallow overparametrized neural network energies, both in the active (aka feature-learning) and lazy regimes. In the active regime, this dual formulation leads to a training algorithm in which one updates concurrently the particles in the sample space and the neurons in the parameter space of the energy at a faster rate. We also consider a variant of this algorithm in which the particles are sometimes restarted at random samples drawn from the data set, and  show that performing these restarts at every iteration step corresponds to score matching training. Using intermediate parameter setups in our dual algorithm thereby gives a way to interpolate between maximum likelihood and score matching training. These results are illustrated in simple numerical experiments.

Under review as a conference paper at ICLR 2022
DUAL TRAINING OF ENERGY-BASED MODELS WITH OVERPARAMETRIZED NEURAL NETWORKS
Anonymous authors Paper under double-bling review
ABSTRACT
Energy-based models (EBMs) generative models that usually trained via maximum likelihood estimation. This approach becomes challenging in generic situations where the trained energy is nonconvex, due to the need sample the Gibbs distribution associated with this energy. Using general Fenchel duality re- sults, we derive variational principles dual to maximum likelihood EBMs with shallow overparametrized neural network energies, both in the active (aka feature- learning) and lazy regimes. In the active regime, this dual formulation leads to a training algorithm in which one updates concurrently the particles in the sam- ple space and the neurons in the parameter space of the energy at a faster rate. We also consider variant of this algorithm in which the particles are sometimes restarted at random samples drawn from the data set, and show that performing these restarts every iteration step corresponds to intermediate parameter setups in our dual algorithm thereby gives a way to inter- polate between maximum likelihood and score matching training. These results are illustrated in simple numerical experiments.
matching training. Using
INTRODUCTION
Energy-based models (EBMs) are explicit generative models which consider Gibbs measures de-
fined through an energy function f, with a probability density proportional exp(- where 3 is the inverse temperature. Such models originate in statistical physics (Gibbs) 2010 Ruelle 1969). and have become a fundamental modeling tool in statistics and machine learning (Wain- wright & Jordan 2008 Ranzato et al.) 2007, LeCun et al.) 2006; Du & Mordatch, 2019 Song & Kingmal| 2021). Given data samples from a target distribution, the learning algorithms for EBMs attempt to estimate energy function to model the samples density. The resulting learned model can then be used to obtain new samples, typically through Markov Chain Monte Carlo (MCMC) techniques.
The standard method to train EBMs is maximum likelihood estimation, i.e. the learned energy is the one maximizing the likelihood of the target samples, within certain function class. One generic approach for this is to use gradient descent, where gradients may be approximated using MCMC samples from the trained model. However, this computationally difficult for highly non-convex trained energies, due to 'metastability", ie the presence of large basins in the energy landscape that trap trajectories for potentially exponential time. This has motivated a myriad of alternative losses to learn EBM energies, such as the popular score matching; see (Song & Kingmal 2021) for review. All in all, such weaker losses come at the expense a loss of statistical power, which motivates exploring computationally efficient methods for EBM maximum-likelihon. estimation.
EBMs also have structural connections with maximum entropy (maxent) models, which have been studied for decades through Fenchel duality. Dai et al. (001bb) was the first work to leverage similar duality arguments for maximum likelihood EBM training. However, their analysis is restricted to en- ergies lying in RKHS balls (i.e. non parametric linear models). Despite the appealing optimization properties of RKHS, these spaces of functions typically only contain very smooth functions when the dimension is large (Berlinet & Thomas- Agnan 2004). A recent line work- oorkinnatin in supervised learning-haa considered an alternative based shallow neural networks (Bach) 2017).

---

Under review as a conference paper at ICLR 2022
which admit linear representation in terms of measure over its parameters and are able to adapt to hidden oow-dimensional structures in the data. The statistical benefits of the obtained Fi or Barron spaces have recently been studied in the context of shallow EBMs by Domingo- Enrich et al. 2021) who show that they may outperform the RKHS models.
In this work, we focus instead on the computational aspects of training such shallow EBMs. Relying on unfnite-dimensional Fenchel duality results for the KL-rrgularized 1,2 and L regression prob- lems over probability measures (App. B), we recast the maximum likelihood training of -BBS into min-max problem over measures, and derive gradient descent-ascent algorithm (Alg. P in the associated metric spaces, based on the Wasserstein distance. Crucially, these schemes, de- fined over idealised parametrisations requiring infinite number of neurons, admit -aitee-article approximation, as in regression or classification. When viewed in terms of particle systems, the dynamics evolve two interacting populations simultaneously: one over the neuron parameters, and the other over the data space (Sec. 3). Moreover, our proposed algorithm naturally interpolates be- tween the primal and dual formualtions, thanks the relative time-scale between the minimization and maximization steps, and is even able to interpolate between MLE and Score Matching. This dual algorithm evaluated experimentally in See.. 5 in a well-calibrated high-dimensional teacher- student environment, which allows us to assess our models against the ground-truth, and test the effect of input dimension. Our experiments confirm that the dual algorithm converges significa- tively faster than the primal one, suggesting that dual updates might bypass metastability despite high-dimensional and non-convex energy landscapes.
Related work. Our work is based on general Fenchel duality results (App. B that may be useful in applications beyond the main focus of this paper (see D). These theorems are a generalization of results stated in the compact case in Domingo-Enm Ennrch et al. (2021) in their Appendix D. Similar the area of maximum entropy (maxent) models (reviewed in Ch. 12 of Mohri et al. (2012)) The first maxent duality principle was due to Jaynes) [1957). Maxent models have been applied since the 1990s in natural language processing and in species habitat modeling among others, and studied theoretically especially since the 2000s (Altun) & Smola 2006 Dudík et al. 2007).
duality results have been studied extensively
Recently Dai al. 2019a) leveraged duality arguments in the context of maximum likelihood EBMs, although in form different from ours. Their duality result works in the more restrictive setting of "lazy" energies lying in RKHS balls and probability measures with L densities, and they derive it directly from a general theorem that works for reflexive Banach spaces (Ekeland & Temam (1999). Ch. 6, Thm. 2.1). Our Fenchel duality results, which work for Borel probability mea- sures and feature-learninn which are non-reflexive Banach spaces. Their algorithm is also different: they do not evolve gener- ated samples, but rather use transport parametrization of the energy. Dai al. (2019b) expand the work (Dai et al. 2019a) combining with Hamiltonian Monte Carlo.
(Fi) energies, more general because must rely measure spaces,
precursor of modern machine learning EBMs were restricted Boltzmann machines (RBMs), first trained via contrastive divergence or CD (Hinton) 2002) which estimates the gradient of the log- likelihood via approximate MCMC samples of the trained model. It later led to maximum likelihood training of EBMs (see e-g. Xie et al. (2016) 2017); Du & Mordatch (2019) among many others). A popular variant of CD is persistent contrastive divergence or PCD (Tieleman) 2008 Tieleman & Hinton| 2009), in which the MCMC samples are evolved and reused to be progressively equilibrated. Drawing a comparison with our work, our dual Fi-EBM training algorithm resembles PCD in that both evolve set of samples over training iterations.
gradient computations
A vast array of EBM losses alternative to maximum likelihood have been developed in recent years (Song & Kingmal (02T) with the goal of avoiding the MCMC procedure, which may be costly for non-convex densities. Some successful ones are score matching (Hyvärinen) 2005) and related methods such as denoising score matching (Vincent) 2011). Building on these, recent works have achieved state of the art image generation (Song& & Ermon 2019 2020 Ho et al. 2020 Song al.) 2021]. We derive the score matching algorithm for F, energies and show a continuum of algorithms interpolating between dual maximum likelihood and score matching training.
2

---

Under review as a conference paper at ICLR 2022
Finally, work (in partieular App. C) also has links with maximum mean discrepancy (MMD)
flows. MMDs are probability metrics that were first introduced in (Gretton et al. 2007 2012) for kernel two-sample tests, and that have enjoyed ample success with the advent of deep- learning based generative modeling as discriminating metrics (Li et al. 2015; Dziugaite et al. 2015 Li et al. 2017).
Among the MMD) literature, the closest work ours is (Arbel et al. 2019). which study theoretically the convergence of unregularized MMD gradient flow (our equation (26) with 3 0). In their experiments, they observe that noisy updates (3-1 > 0) needed for good generalization. Our work shows that their algorithm is exactly training maximum likelihood EBMs energies in an RKHS ball of radius that depends on the noise level (see App. Cp.
2 BACKGROUND AND SETUP
In this section, we provide preliminary background on the functional spaces associated to over- parametrized two-layer networks, and on EBMs and their training losses.
Notation. V is a normed vector space, we use Bv (B) to denote the closed ball of V of radius 3, and By := (1) for the unit ball. If K denotes subset of the Euclidean space, P(K) is the set of Borel probability measures, M(K) is the space of Radon (i.e. signed and finite) measures, and (K) is the set of non non-negative Radon measures. If r is a Radon measure over K, then is the total variation (TV) norm of % which turns M(K) into a Banach space. Throughout the paper, and unless otherwise specified, a : R + IR denotes a generic non-linear activation function. We use (+)+ R - to denote the ReLU activation, defined as (z)+ m2. We use to denote fixed base probability measure, possibly used with a subindex to specifyy the space it is defined over. We use Sd S gd+1 for the d-dimensional hypersphere and log for the natural logarithm. We denote the Lebesgue measure by X. Given two probability measures V, v/ E PKK)) DKL (xL(v/l/)) =
Iblll = J d|
de du denotes the KL divergence from v/ to v and = Sx log dul dr )dv is the cross-entropy.
Hu,,)) - Jxx log( de'
2.1 OVERPARAMETRIZED TWO-LAYER NEURAL NETWORK SPACES
In this work, we will focus on dense function approximation classes generated by overparametrized shallow neural networks. One can distinguish two canonical models, depending on the asymptotic scaling regime. For further background on these regimes, we refer the reader to Chizat et al. (2019).
Kernel regime. Let X' C Te O S Re y : x θ - TR, and re be a fixed base prohability mea- sure over θ We define F, as the reproducing kernel Hilbert space (RKHS) of functions f X - IR such that for some h E I.2 (,) re), we have that, for all x E X, f(x) So sax () (0). The
RKHS norm of F2 is defined as [l,fl>, = inf |||]| (e(e)) I f(-) = J sp(,, 0) ((6 df) } where
[[b] == Jo [2())12 (f) (c.f. Bach (211))
). As an RKHS, the kernel of Fg is
k(x,y)= = [ p(x, 0))(,, 0) (o)
(1)
Feature-learning regime. Set X. θ and to as in the previous paragraph, and define Fi as the Banach space of functions f X + R such that, for some Radon measure Y MM()),
for all x E X we have f(x) = (ex 0) do). We define the norm of Fi as ||
inf { |I>iTT lf(-) fo p(, 0) d>(0)] This construction was introduced by Bach 2017) who first
used the notation Fi and focused in particular on the case X' C Rdd θ = Sa and (0xx, 0)
ReLu"(((e) 1), ))) for some & E Z.. This space is also known by the name of Barron space (D
ct al. 2019 Wojtowytsch 2020 reference to the classic work (Barron 1993).
Remark that since [[ = So h(()) (T)(0) S (es [0)) (0) = [ by the Cauchy-Schwarz inequality, we have F2 C Fi: in particular finite-widte width neural networks belong
3

---

Under review as a conference paper at ICLR 2022
to Fi but not to F2 Bach 2017 The TV norm in Fi acts pparsity-promoieee penalty, which encourages the selection of few well-chosen neurons and may lead to favorable adaptivity properties when the target has now-dimensional structure.
2.2 EBMS AND TRAINING LOSSES
Consider a measurable set X' S Rd with fixed base probability measure Tx E P(A). is class of functions (or energies) mapping X' to R, for any f E J we can define the probability measure Vy as a Gibbs measure with density:
dvps (x) = 13 -8(xx) drx
Zøs == | (-Bf())dx(())
with
where dugs/ /dxx is the Radon-Nikodym derivative of Var and Zzz is the partition function. The parameter is the inverse temperature. We could merge into F by considering the function class {f E Fj. but we have decided to keep them separate to showcase the dependency on 3. Gibbs measures are the cornerstone of statistical physics since the seminal works of Boltzmann and Gibbs. Beyond their widespread use across computational sciences, they have also found their application in Machine Learning, by the name of energy-bused models (EBMs), where the energy function is parametrised using e.g. neural network.
We denote by Fi-EBMs the energy-based models for which the energy class F is the unit ball By, (1) of Fi. Notice that the class {ff)f FF} is equal the ball B5, (5). Such models may be regarded as abstractions of more complex deep EBMs, in that they incorporate feature learning, and they were first studied by [Domingo-Ennie They are to be contrasted with F2-EBMss for which F is the unit ball By, (1). F2-EBMs, which we study in App. C have fixed features and showed worse statistical performance experiments (Domingo- Ennich et al. 2021).
et al. (202T). which provide statistical guarantees.
Given samples { from a target measure Up, training an EBM consists in selecting the best Paf with energy according given criterion. Two such criteria relevant in this paper.
Maximum likelihood. The maximum likelihood estimator (MLE) is defined as f arg yax IT drx (zi), or equivalently, as the minimizer of the cross-entropy H (vn, vgs) with the empirical measure Un = = +I= òz::
ddgs (1:) = arg min
1 71
= arg min JEF
log
Σ fxx) + 1- llg Zzz-
(2)
drx
JEF
drx. By observing that DKL (vlle)) = H(v,v') - Hv),, where ((v) == H(, is the entropy, minimizing the cross- entropy is equivalent minimizing the KL divergence when the latter is finite. However, such equivalence is only well-defined the population setting where the empirical measure V, re- placed by its population counterpart pp- An appropriate choice of function class F induces reg- ularization that prevents the learned Gibbs measure to overfit to the empirical data measure, and presumably approximate V, instead. The MLE enjoys strong statistical properties (Wainwright & Jordan 2008; Wainwright (219)), as well as a powerful variational principle as soon as one consid- ers convex function classes (see Appp. A), but its notorious computational challenges (see Related works section) have motivated alternative approximation metrics to be used to learn EBMs.
The estimated distribution is then simply given by dvgs
38 dTxx.
Since an arbitrary element f of Fi can be expressed as f(x) = Se s(z, 0) dr((). with lb= equal to the infimum of ||vl for all such the maximum likelihood function for F = =By, (1) the problem (2 can be restated as fMLE Sa (((=0))MMME()) where
1 p( 8) d0) log (/e (soxp// (-9 plx, dr(0) (((,)))((()) drx(ot) (()
Σ
/ 1=1
MMLE = arg min
rEM(e) n
4

---

Under review as a conference paper at ICLR 2022
Score matching. An important instance of such weaker metrics is given by Score Matching (SM). The SM metric between two absolutely continuous probability measures 1,, is defined as SM(P, v') = Jx VV log drx (x) V log del (2)]² dv(x). The SM metric is known in information theory as the relative Fisher information. Note that this metric cannot be trivially extended to the case v = Un, because empirical measures do not have density with respect TX. To get around this difficulty, note that, if the target measure Vp is absolutely continuous with respect to Tx and (ar) its energy, learning EBM with function class F under the population loss corresponds to solving f arg min fer fx [V(()) ffp(x)) dz)(2). The insight from Hyvärinen (2005) is that under regularity conditions on fp- via integration by parts we then have fx |V/ V 5 dup = [2,,); L(J, vn) + C, where Ra.. denotes expectation over the data set, C' is constant in f which therefore irrelevant, and
du
denote by [p(x) -3-1 log dvz drx
+ L(f.vm) = 1 + + (V.(E))
In practice, we train an EBM via score matching by solving / arg min L(f, un)- Score matching is computationally more tractable than maximum likelihood and performs well in practice (see Related works section). Statistically, its main drawback is that the SM metric weaker than the KL divergence, and may fail to distinguish distributions in some instances.
The following proposition, proved in App. G provides the expression for the loss L and the resulting score matching problem for F.BEBMs.
Proposition 1. Suppose that X S Rd is manifold without boundaries. Assume that du (z)] drx(x) is upper-bounded by some constant K for all 0 A. Assume also that supage |l ( 0) < n(xx) and that Sx |n(x)] dvy(x) < cc. The optimization problem to train EBMs under the score matching loss over the ball By, (1) gives fsM = Sa () where
Sx Vupp(z, 0) drx
1 (EVoplo,)) V. I 0) dy(0)) B-' ('sppp(,00) 0)) du (2) 7EM19) IillvsSI
YSM = arg min
DUAL FBBE TRAINING VIA MAXIMUM LIKELIHOOD
As corollary of the duality result from Subsec. B.2 we derive an alternative objective for -BBMS trained maximum likelihood, the original objective being (3) and we develop an algorithm to solve this alternative problem. To this end, we make:
X' x θ + R be a continuous function such that either X' is compact or for any fixed O, sp(x, E(x:) for some strictly positive X RR and (ūi E(x) log(E(x))
Assumption 1. Let so
log
drx dx (x+)
(di + log [1rlla) as ||0||22 + to0 for some 0.
In particular, this assumption holds for ReLU network energies when setting X' Rd, = Rd++1 (0x 0) = (((r,, 1), / and xx Gaussian (and in many other settings). Theorem 1. Under Assumption 1. the problem 3 is the Fenchel dual of
min max 8 Dxe(lTlTT + ( ple, d(y - vn)(x) dy(0). MEP(X)
(5)
(e)))),
Moreover, the solution V² of (70) is precisely the Gibbs measure for the optimal > in (3) that is,
du (x) z exp (-3 ((x, dy (x)).
drx
we replace the F, ball by the F2 ball, the analogous duality result links the maximum likelihood problem with the entropy regularized MMD flow from Arbel et al. (2019) (see App. C).
5

---

Under review as a conference paper at ICLR 2022
Training dynamics nonnegative measures. Let us write the dynamics solve that can be discretized in terms of parameters and particles (effrooosiiion 2)below). To this end we consider the triple (y+, ? 1/) where the nonegative measures 7 are defined through the Hahn decomposition of y y= -------- 7- Then we introduce coupled gradient flows for this triple, in which y and 7 evolve via a Wasserstein- -enesteii--isher--ee Rao gradient flow (Chizat et al. 2018) and i evolves via a Wasserstein gradient flow (Santambrogio 2017]
O VaoVo VoF((O) on (oFi(t) Ki),
o =1+
7i =
0.v: = V. Vali(()) -
V log dTx
dA )) + 1-8 +8--Aaus
(6)
where a is a tunable parameter and we defined
Ot) [ (pxx 0) ((lt-va((a)))
f((x) ((x, 0) (dyt (0),
Kr 1 Inv 111v21 (.(0)(dT) (0).
The initialization of (6 is vo = Vn and 70 0 (such that the initial energy is null). The term K, keeps the total variation of 7/2 below one. The parameter a acts relative timescale. Notice that different values of a can potentially lead to different behaviors of the dynamics; setting a < would correspond the primal formulation of maximum likelihood with persistent MCMC samples (as in PCD). In contrast if a >> 1, T evolves faster than V/ and if the optimization well behaved at all times 7/2 = % % remains close to minimizing the inner maximization problem of (5) with y yt. Initializing V/ vn, is crucial avoid the kind of metastabilitics that curse the behavior of classical (primal) maximum likelihood EBM training.
below states that the solution (He, may be approximated using coupled particle systems (see proof in App. F) and is the basis for Alg. [ The link between particle systems and measure PDEs is through a classical technique known as propagation of chaos (Sznitman) [991] and it has been used previously for similar coupled systems in the machine learning literature (Domingo-
Proposition 2
Enrich et al. 2020).
Proposition 2. Let ( be initial features sampled uniformly over θ let {oz be uniform samples over }+ and let (wog = 13} be the initial weight values, which are set to I. Let IN JN be the initial "generated" samples, which are chosen i.i.d. uniformly from the target sample set {xr) Consider the system of ODEs/SDEs:
do(3)
dwl)
dt = ao, L (..
dt = am (o, -Ki)
(8)
ax(()) = (-x((x(()) +-8-1 - log drx (x())) dt + /28-1 aaw
dd
where
1 n n
1 m
P((0) Σ
x(x (
.0
plx; 0),
f((x)
Σ j=l
(1) og u plz, 0(0))
K, = Em m
aj
(O())))
the empirical counterparts of the functions in Then the system dynamics. Namely, as m, N DC:
8 approximates the measure
the empirical measure &: - E w 8 وeو converges weakly to the solution 7/t = 7t Ve
of (6) with uniform initialization for any finite time interval [0, T), and
the empirical measure ir = + -i 8 8x10) converges weakly to the solution Vi of 6 for any
finite time interval [0, ().

---

Under review as a conference paper at ICLR 2022
Importantly, the system of ODEs/SDEs in ( may be solved via forward Euler steps on {8,] and ) I makes use of a tunable parameter PR. which stands for the restart probability and will be discussed in [Sec. 4 as natural way to connect maximum likelihood with score matching. To discretize
{w;) 1 (or rather, {log w; 1 ) and Euler-Maryyama updates {X,0) Ji=1 Such discretization yields Algorithm[I] [T] Algorithm
(8) we set pn = 0, i.e., there are no partiele restarts.
Algorithm Dual Fi FB- training (pR O: maximum likelihood, pr (saA)); score matching)
Input: n samples { of the target distribution, stepsize 8, stepsize ratio a. >ym in [0, 1), signs (a,) over {+1). Initialize generated samples (X)(") Jim1 uniformly i..d. from {a:]--.. for t = 0,..,, for i ................................ N do
Initialize unif. features (0()))² >5= over 6, weights (wo)) )0m
do
With probability pR. replace x by some uniformly chosen sample in (x:)²=1 (see Sec. 4
Sample ((()) from the variatte standard Gaussian.
Perform Euler-Maruyama update: X = x xVVi(( + 8 log dex
(X( +
Ax
V 128-18 c( where fe is defined in end for for
I...., .......7... m do
Update 0(1) 8(5) + soØ, VF.(O)).
) where F, is defined in 9
1+1
Update ( = (i) (xpssaa,F:(0)")
Normalize needed w = a(1) max (m- En th (01)
"(1111 1).
end for end for
Output: samples (x Jimi energy fr(x) = m -m o,
LINKS BETWEEN MAXIMUM LIKELIHOOD AND SCORE MATCHING F--BBMs
In this section we uncover how the score matching loss fits seamlessly variant of Alg, 1 in the form of particle restarts. Interestingly, we can modify the PDE (6) in a way that allows us to make connection with score matching. To this end, let introduce the following coupled measure PDE:
O.y --vo V. VoF.()) + an? (aFx(0) K2)
a == #1, %e T
Mt V (zf((x) 3 \oolg dTx dx ))
(10)
0.4, Vx
log
+ 3-
A +
a (vz - un).
Remark that the only difference between this equation and the PDE (6) for dual maximum likelihood training is the term -a(u;- -vz), which draws U, closer to the empirical target measure Vy- We have:
Proposition 3. In the limit a - co, the equations for 78 in 10 reduce to
Ocf = VV (TE V V (T:)()))))))))))))))))))))))))))
VVV
(
a=
Ve = (1I)
where = --- - % (n) = lo (n)dd,, and (() is the Frechet derivative of the score matching loss L M(() + IR defined in (4.
That is. in the large a limit, equation 10 is equivalent to the Wasserstein- Fisshrr---iiaaaaa gradient flow of loss L. which, remarkably, is the score matching loss for FB-BMMs. This means that adding the term -a (vz - v,, to the dual maximum likelihood measure dynamics and letting a - oo we recover the score matching dynamics. This additional
can be easily implemented at particle level by replacing each training sample x(" by some random target sample in {z, 1r, with probability pn- 1 -at at + o(t) for every time interval of length L (proof in App. G
7

---

Under review as a conference paper at ICLR 2022
Similar birth-death processes used in (Rotskoff et al., 2019) in the context of neural network regression. Hence, the score matching scheme corresponds to setting the restart probability pR = so in Algorithm|] [ The restart probability acts as knob that allows us to interpolate between score matching and maximum likelihood.
In summary, score matching differs from dual maximum likelihood in that the trained measure being "pulled" towards the target measure at all times via particle restarting. Such constant pulling should be useful to alleviate sampling problems due to metastability issues which may arise with dual maximum likelihood. However, dual maximum likelihood has the upside of providing samples of the learned EBM as a byproduct of training, which score matching does not. A good balance between both algorithms may be use restart probability pa between that decreases with time from one value to the other, in such a way that at the beginning of training we avoid metastability issues by restarting the particles frequently, and an advanced phase we perform little to no restarting to obtain faithful samples. is also interesting to contrast our approach to score matching with the works Sutherland et al. (2018); Arbel & Gretton (2018), which using different techniques propose algorthmms to train EBMs with RKHS energies via score matching, Finally, notice that a particle discretization of the flow (TI) yields an alternative straightforward algorithm to train Fi Fs via score matching; see Subsecc G.I] In Subsec. G.I we show that this algorithm can be linked directly to Alg. with particle restarts, without recurring to measure arguments.
A 1 and O, even a Pr
5 EXPERIMENTS
Setup. To illustrate Alg. [we perform numerical experimentss on simple synthetic datasets generated by teacher models with energy f*(x)= - E/= (()) z)), with f3 E St for all j. The training is performed using Alg. I with the added detail that both the features o(5) and the particles x(( are constrained to remain on the sphere by adding a projection step in the update of their positions. The code, figures, and videos on the dynamics can be found in the supplementary material. In the main text we consider two planted teacher neurons (J = 2) with negative output weights wj = w = 64 neurons for the student model, but we include additional
-10 in dimension d 14 and experiments and videos in App. and supplementary material. We study setups with two different choices of angles between the teacher neurons, which showcase different behaviors:
Teacher neurons 87 03 forming angle of 2.87 rad (< 164 degrees), and output weights w w = 10. The teacher neurons are almost in opposite directions, and the resulting target distribution is bimodal, the energy has two local minimizers around 0ī and 03 (see Figure 5).
Teacher neurons 07 f8 forming an angle of 1.37 rad (s 78 degrees). The teacher neurons are almost orthogonal and the resulting target distribution is monomodal; indeed, when the angle is less than /2, the target energy has unique minimizer at the geodesic average between 0 and 85 (see Figure in App. lp.
Monitoring convergence. In all our experiments, monitor convergence use testing set of data points sampled from the teacher distribution: denoting these samples by { V we estimate the KL divergence from the student the teacher via log( Σn exp(-$f. + (x))) + / E (fi(x)) + (xz)) where fe(x) = m E/ w. the estimate - Σn (af((a;)- (x))
(1) o((( xx)). Similarly, for the score
matching objective
Comparison of the primal algorithm and the dual algorithms. We defer the empirical study of tuning the restart probability to App. and in this section focus on comparing the dual algorithm for maximum likelihood Fi-BBMs (i.e. with PR = 0) to the classical (primal) algorithm, which was the algorithm used in the experiments of Domingo--Ente corresponds to Alg. with a < 1, while the dual algorithm uses & > I. To obtain a principled comparison of the two settings where numerical errors do not blow up, set to be the step- size for the fastest process (particle evolution for the primal, neuron evolution for the dual), and min(a, 1/a)s the stepsize for the slow process.
et al. (2021). The primal algorithm
8

---

Under review as a conference paper at ICLR 2022
100
10
00.5
KL div
=0
SM
10
5
10
100
0
500
1000
1500
2000
0
500
1000
1500
2000
500
1000
1500
2000
SGD iteration
SGD iteration
SGD iteration
10
10
= 0.05
KL div
20
SM
2 5
10a2
10
5000
10000
5000
10000
5000 SGD iteration
10000
SGD iteration
SGD iteration
Figure 1: (Top) The evolution of the KI. divergence, the score matching metric and the TV norm of the trained measure (i.e., the Fi norm) during training for Algorithm | with X' = S m = 64, PR 0, 0.02, n 105, N 2- 10 and a 0.05 (primal training) or a = 20 (dual training). showing speedup by factor about 10-20 of the latter over the former. The angle between the two teacher neurons 1.37 rad (monomodal distribution). (Bottom) Same experiments with an angle of 2.87 rad between the two teacher neurons (bimodal distribution).
The results are shown in Figure 6) for the two angle configurations between teacher neurons. We observe that the dual algorithm several orders of magnitude faster at reaching KL and SM val- ues close to the final ones, which showcases the main advantage of the dual approach. For the monomodal distribution, the final values obtained by the dual algorithm are slightly better than for the primal algorithm; for the bimodal distribution, the converse happens and the convergence slower for both algorithms, most likely due to metastability. Interestingly, the decrease of the performance metries seems stall soon the hard Fi-norm threshold is reached.
DISCUSSION AND OUTLOOK
In this work we leverage Fenchel duality result to recast the maximum likelihood loss for Fi- EBMs into a min-max problem on probability measures over the sample space. We provide mean- field dynamics at the measure level to solve this problem, which lead to dual algorithm (Alg. [ after discretization. We observe that we restart particles at random target samples throughout training, get algorithm which is equivalent to training under the score matching loss. We perform experiments in which we learn planted distributions with two-layer ReLU networks, and observe empirically that our dual algorithm is much faster than the classical one.
At theoretical level, one direction for future work is obtain convergence results for the dynamics (6) and (TO). Domingo-Ennn Enrich et al. (2020) study similar coupled Wasserstein-Fishers flows, but their results only work in the of weight learning rates much larger than position learning rates. We hypothesize that the additional term -a (vt vn)) which keeps 1/2 close to Vn. might help in the analysis.
gradient
At the numerical level, it would be interesting to further test the variant of Algorithm with an- nealed pn decreasing time, to understand under which parameter setup captures the best fea- tures of maximum likelihood and score matching. One could also test Algorithm I using deeper neural architectures: while the analysis is more complicated this case, the scheme itself can be straightforwardly generalized deep networks.
REFERENCES
Yasemin Altun and Alex Smola. Unifying divergence minimization and statistical inference via convex duality. In Learning Theory, pp. 139-153. Springer Berlin Heidelberg, 2006.

---

Under review as a conference paper at ICLR 2022
Michael Arbel and Arthur Gretton. Kernel conditional exponential family. In Proceedings of the Twenty- First International Conference on Artificial Intelligence and Statistics, volume 84 Pro- ceedings of Machine Learning Research, pp. 1337-1346. PMLR, 2018.
Michael Arbel, Anna Korba, Adil Salim, and Arthur Gretton. Maximum mean discrepancy gradient flow. In Advances in Neural Information Processing Systems, volume 32, 2019.
Francis Bach. Breaking the curse of dimensionality with convex neural networks. Journal of Ma- chine Learning Research, 18(19)11-53, 2017.
Andrew Barron, Universal approximation bounds for superpositions a sigmoidal function. Infor- mation Theory, IEEE Transactions on, 39:930 945, 1993.
Alain Berlinet and Christine Thomas-Agnan Reproducing Kernel Hilbert Space in Probability and Statistics. Springer, 2004.
Jonathan Borwein and Qiji Zhu. Techniques of Variational Analysis. CMS Books in Mathematics. Springer- Verlag New York, 2005.
Zhengdao Chen, Grant M. Rotskoff, Joan Bruna, and Eric Vanden-Eijnden. A dynamical central limit theorem for shallow neural networks, 2020.
Lenaic Chizat and Francis Bach. On the global convergence of gradient descent for over- parameterized models using optimal transport. In Advances in neural information processing systems, pp. 3036-3046, 2018
Lénaïc Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Lénaïc Chizat, Gabriel Peyré, Bernhard Schmitzer, and François-Xavier Xavier Vialard. Unbalanced opti- mal transport: Dynamic and kantorovich formulations. Journal of Functional Analysis, 274(11): 3090-3123, 2018
Dai, Hanjun Dai, Arthur Gretton, Song, Dale Schuurmans, and Niao He. Kernel exponential family estimation via doubly dual embedding. In Proceedings of the Twenty-Second Interna- tional Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine PMLR, 2019a.
Learning Research, pp.
Bo Dai, Zhen Liu, Hanjun Dai, Niao He, Arthur Gretton, Le Song, and Dale Schuurmans. Exponen- tial family estimation via adversarial dynamics embedding. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019b.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. arXiv preprint arXiv:2105.05233, 2021.
Carles Domingo-Enrich, Samy Jelassi, Arthur Mensch, Grant Rotskoff, and Joan Bruna. A mean- field analysis of two-player zero-sum games. In Advances in Neural Information Processing Systems, volume 33, pp. 20215-20226. Curran Associates, Inc., 2020.
Carles Domingo-Enrich Alberto Bietti, Eric Vanden-Eijnden, and Joan Bruna. On energy-based models with overparametrized shallow neural networks, 2021.
Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-based models. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Miroslav Dudík, Steven J. Phillips, and Robert E. Schapire. Maximum entropy density estimation with generalized regularization and an application species distribution modeling. J. Mach. Learn. Res., 8: 1:12 2007.
Nelson Dunford and Jacob T Schwartz. Linear operators. Part I, General theory. Pure and applied mathematics (Interscience series). Interscience, 1958.
Gintare Karolina Dziugaite, Daniel M Roy, and Zoubin Ghahramani. Training generative neural networks via maximum mean discrepancy optimization. UAI, 2015.
10

---

Under review as a conference paper at ICLR 2022
Weinan E and Stephan Wojtowytsch. On the banach spaces associated with multi-layer relu net-
works: Function representation, approximation theory and gradient descent dynamics, 2020.
Weinan E, Chao Ma and Lei Wu. A priori estimates of the population risk for two-layer neural networks. Communications in Mathematical Sciences, 17:1407-1425, 2019.
Ivar Ekeland and Roger Temam. Convex analysis and variational problems. Philadelphia, Pa:
Society for Industrial and Applied Mathematics, 1999.
Josiah Willard Gibbs. Elementary Principles in Statistical Mechanics: Developed with Especial Reference the Rational Foundation of Thermodynamics. Cambridge Library Collection Math- ematics. Cambridge University Press, 2010.
Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alex Smola. A ker- nel method for the pp. 513-520 520, 2007.
In Advances in neural information processing systems,
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(25):723773, 2012.
Geoffrey E. Hinton. Training products experts by minimizing contrastive divergence. Neural Comput., 1778)1177--1800, 2002.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33. Curran Associates, Inc., 2020.
Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(24):695-709, 2005.
E. T. Jaynes. Information theory and statistical mechanics. Phys. Rev. 106:620-630, May 1957.
Alexia Jolicoeur-Martincau, Rémi
Rémi Tachet des Combes, and Ioannis Mitliagkas. Adversarial score matching and improved sampling for image generation. arXiv preprint arXiv:2009.05477 2020.
Zahra Kadkhodaie and Eero P Simoncelli. Solving linear inverse problems using the prior implicit in denoiser. arXiv preprint arXiv:2007.13640, 2020.
H. Kneser. Sur theoreme fondamentale de theorie des jeux. C. R. Acad. Sei. Paris, 234: 2418-2420, 1952.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. A tutorial energy learning. 2006.
Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabas Poezos. Mmd gan: Towards deeper understanding of moment matching network. In Advances in Neural Information Processing Systems, volume 30, 2017.
Yujia Li, Kevin Swersky, and Richard Zemel. Generative moment matching networks. ICMI., 2015.
Henry McKean. A class of markov processes associated with nonlinear parabolic equations. Pro- ceedings of the National Academy of Sciences of the United States of America, 56:1907-11, 01 1967.
Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two- layer neural networks. Proceedings of the National Academy of Sciences, 115633):7665-E7771, 2018.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning. The MIT Press, 2012.
J Neumann. Zur theorie der gesellschaftsspiele. Mathematische annalen, 100(1):2953320, 1928.

---

Under review as a conference paper at ICLR 2022
Edward C. Posner. Random coding strategies for minimum entropy. IEEE Transations Informa- tion Theory, 21(():388- 1975.
Marc Ranzato, Christopher Poultney, Sumit Chopra, et al. Efficient learning of sparse representa- tions with energy-based model. 2007.
Grant M Rotskoff and Eric Vanden-Bijnden. Neural networks interacting particle systems: Asymptotic convexity of the loss landscape and universal scaling of the approximation error. arXiv preprint arXiv:1805.00%: 2018.
Grant M Rotskoff, Samy Jelassi, Joan Bruna, and Eric Vanden-Fijndes Global convergence of neu- ron birth-death dynamics. In Proceedings of the 36th International Conference on International Conference on Machine Learning. Long Beach, CA, USA, 2019.
D. Ruelle. Statistical mechanics: Rigorous results. W.A. Benjamin, 1969.
Filippo Santambrogio. (Euclidean, metrie, and Wasserstein] gradient flows: an overview. Bulletin of Mathematical Sciences, 7(1):77-154, 2017.
Maurice Sion. On general minimax theorems. Pacific J. Math., 8(1):171-176, 1958.
Justin Sirignano and Konstantinos Spiliopoulos. Mean field analysis of neural networks: A central limit theorem. Stochastic Processes and their Applications, 2019.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. arXiv preprint arXiv:990..05600, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. In Advances in Neural Information Processing Systems, 2020.
Yang Song and Diederik P. Kingma. How to train your energy-based models, 2021.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations (ICLR 2021), (, 2021.
Danica J. Sutherland, Heiko Strathmann, Michael Arbel, and Arthur Gretton. Efficient and princi- pled score estimation with nyström kemnel exponential families. In Proceedings of the Twenty- First International Conference on Artificial Intelligence and Statistics, volume 84 of Proceedings of Machine Learning Research, pp. 652-660. PMLR, 2018
Alain-Sol Sznitman. Topics in propagation of chaos. In Paul-Louis Hennequin (ed.), Ecole d'Eté de Probabilités de Saint-Flour XIX 1989, pp. 165-251, Berlin, Heidelberg, 1991. Springer Berlin
Heidelberg.
Tijmen Tieleman. Training restricted boltzmann machines using approximations to the likelihood gradient. In Proceedings of the 25th International Conference on Machine Learning. ICML '08. Association for Computing Machinery, 2008
Tijmen Tieleman and Geoffrey Hinton. Using fast weights to improve persistent contrastive diver- gence. In Proceedings of the 26th Annual International Conference Machine Learning, ICMI. '09. Association for Computing Machinery, 2009.
Pascal Vincent. connection between score matching and denoising autoencoders. Neural Compu- lution, 23(7), 2011.
Martin Wainwright and Michael Jordan. Graphical models, exponential families, and variational inference. Foundations and Trends in Machine Learning, 1:1-305, 01 2008
Martin J. Wainwright. High- fighe-Dimessoonll Statistics: A -on-Asyyptooiic Viewpoint. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2019.
Jianwen Xie, Yang Lu, Song-Chum Zhu, and Yingnian Wu. A theory of generative convnet. In Pro- ceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research. PMLR, 2016.
12

---

Under review as a conference paper at ICLR 2022
Jianwen Xie, Song Zhu, and Yingnian Wu. Synthesizing dynamic patterns by -empiall generative convnet. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
13


Reviews:
[{"review_id": "ssdg9rnIMle", "reviewer": "Reviewer_hfDM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a theoretically rich paper. I am not able to provide insightful technical comments in this urgent review, and I think the contributions in the main text and appendix are interesting though the significance is less well demonstrated. I personally find this paper hard to follow, with subtle unclear statements and jumps at various places.\n\nClarify:\n1. What is meant by \"active\" and \"lazy\"? There are only two mentions of \"active\" and and three \"lazy\" in the combined main+appendix supplementary file. If these are stated in the abstract, they need to be clearly specified in the main. \n2. The following statement: \n> \"Such models may be regarded as abstractions of more complex deep EBMs, in that they incorporate feature learning,\nand they were first studied by Domingo-Enrich et al. (2021), which provide statistical guarantees.\nWhat's meant by statistical guarantees? Statements like this make readers second guess what the authors want to convey. [The ref of this highly related paper indicates that the paper is not yet published, but it is actually accepted in May and published in July 2021, well before the current submission deadline.]\n3. There is a huge jump from (5) to (6), which I cannot follow, with only 8 lines of text explanations. What's $K_t$?\n4. The remark at the end of page 3: what's $L1(Θ)$ norm? Is this related to the TV norm?\n\nExperiments:\nThe experiment demonstrates the simplest possible setting, which I reckon is of little interest to most readers of this conference. Most other cited and theoretically rich papers demonstrate at least on a few real datasets. In particular, no experiments try to verify the Proposition 3, one of the more interesting theoretical results in my opinion. This makes it hard to support this paper in terms of its significance. \n\nMinor issue:\nFor the state results on score matching to hold, the definition of the score matching (relative Fisher information) should be \ndefined with a factor of 1/2 in the proof to Proposition 1.", "summary_of_the_paper": "The authors formulated the Fenchel dual problems to maximum likelihood and score matching training applied to energy-based models defined by shallow neural networks. The authors then proposed practical algorithms based on the Hahn decomposition and compared this to various predecessor algorithms. An interesting part is that the learning dynamics can interpolate between maximum likelihood and score matching. There are also rich results in the Appendix. ", "summary_of_the_review": "This paper potentially has rich and interesting theoretical insights, but the results are very poorly described and insufficiently demonstrated in experiments. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "kJTiU6-GFk9", "reviewer": "Reviewer_aaPp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n* As far as I know, the paper is self-contained and technically sound. The author introduces the motivation and derivation in detail. The proposed algorithm naturally connects MLE and SM.\n* Numerical experiments seem to show that it is a promising alternative to train EBMs.\n\nWeaknesses:\n* I would say that the notation and formulation are a bit heavy and not familiar to the ML community, which makes the paper really hard to follow. I have to spend quite a long time reading through the paper and getting the points.\n* The paper empirically shows that the dual algorithm trains the EBM faster. However, I would like to see more theoretical analysis in terms of the convergence speed. \n* The numerical experiments are all based on synthetic datasets generated by predefined EBMs. I would like to see at least one example of applying the proposed algorithm to a realistic dataset (e.g., MNIST). ", "summary_of_the_paper": "This paper derives a Fenchel duality formulation of the maximum likelihood loss of F1-EBMs, which turns the optimization into a min-max problem on probability measures over the sample space. A dual algorithm with mean-field dynamics is proposed to solve this problem. Using the dual aogrithm they also draw a connection between maximum-likelihood training and score matching. Simple numerical experiments on two-layer ReLU network show that the algorithm converges much faster than the original MLE. ", "summary_of_the_review": "Overall speaking, this is a decent and novel paper analyzing the training algorithm of EBMs from the perspective of probability measure. The paper could be strengthened if more theoretical analysis of convergence speed and more empirical results are included. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7mL3xbPMVXy", "reviewer": "Reviewer_RMY3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Let me start by saying that I strongly agree this energy based model is useful in many aspects of machine learning. But the key question I find myself asking is what the practical benefit of the proposed method is compared with scoring matching, and what difficulty it bypasses in the original MLE problem.\n\nThe author does not explain too much on why this dual problem is necessarily better than the primary problem other than saying the latter can be metastable. Below is my take: The MLE problem (3) is hard because of the difficulty of computing the partition function $Z_{\\beta}$, while this task disappears in scoring matching. The dual problem (5) does not need to compute the partition function, but sill need to compute very similar high-D integral (the second term). At a very high level, (5) is akin to use adaptive-importances-sampling/bride sampling to compute $Z_{\\beta}$, while we also optimize the proposal (base) density simultaneous on the fly.\n\nYet even it is much better than the primary problem, why is it better than the score matching (4), in which there is no need to compute  $Z_{\\beta}$ at all? The experiment section is weak and there is no empirical evidence to support this claim. Yes, score matching can be derived as a special case by letting $\\alpha \\to \\infty$, but score matching is much easier to train even without the proposed dynamics.\n\nHow is $\\alpha$ tuned? How sensitive is the method to $\\alpha$?\n\nThe structure of the paper is fine. But I am wondering is the introduction of $\\mathcal{F}_1$ and  $\\mathcal{F}_2$ totally relevant to the method development? Although I understand they imply two extremes of lazy and active learning, I don't find this difference is critical to the proposed method in the practical level.\n\neq 1 seems to be a typo.\n", "summary_of_the_paper": "Two approaches exist for learning a Gibbs measure: either by MLE (minimizing the KL divergence) or by score matching (maximizing the Hyvarinen score). With the same goal to minimizing the KL divergence, the author proposes an alternative training approach by augmenting an auxiliary $\\gamma$ measure and thereby optimizing a more tractable dual problem. The author claims that this new training method has a quicker convergence rate than the primary problem.  ", "summary_of_the_review": "The proposed method is novel and promising for some useful applications. The paper is tightly presented although some reorganization can make the main contribution clearer. However, the theory justification, comparisons to the benchmark, and empirical evaluations are weak.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-3LOmmntLzB", "reviewer": "Reviewer_pG2E", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The connection on a continuum of maximum likelihood and score based training is very interesting from a theoretical point of view, and I think this is a novel and valuable contribution. \n\nMy biggest concerns about the work are the following: \n\n1) In the introduction it is stated that score matching is a weaker loss than maximum likelihood or, in other words, that score matching falls short in statistical power. However, the empirical evaluation in the paper shows that alpha>>1 (Score Matching) is superior to alpha<<1 (Maximum Likelihood). Isn’t this a contradiction? How do the authors explain this phenomenon? In summary I am missing a discussion along the following lines: “while Score Matching is weaker than Maximum Likelihood, our theoretical analysis shows that the impact on the convergence of alpha is …, and consequently we can explain why alpha >> 1 is better in practice”.\n\n2) While I really appreciate the theoretical study in this paper, what I really miss is an implementation of the idea outlined at the end of section 4, which is what the authors refer to as an annealed p_R. This is exactly what the theoretical method needs: it enables (potentially) the development of a new algorithm that bounces from score matching to maximum likelihood, and achieves the best of both worlds. Such an algorithm could only have been conceived and theoretically justified with the work presented in section 3 and 4.\nThe “sentiment” I have is that the paper in its current form is not finished, because there is a disconnect between the theory and the empirical findings.\n\n3) The experimental validation is somehow weak, as only a simple synthetic dataset is considered. If it wasn’t too complicated (as the authors suggest in their conclusion) to port Algorithm 1 to a deeper model (without a theoretical analysis, of course) why didn’t they do it?  Combining with point 1) above, it is hard to extrapolate any valid claims with a single and simple example. What guarantees do we have that the alpha>>1 would be better also in other settings? While it is not useful to explore deeper models, or for example begin with a comparison of the considered architecture (a 2 layer Neural Network) on standard datasets like UCI? This would clearly strengthen the paper.\n\nMinor: \n1) It is not clear why the discussion about the two different F_1 and F_2 spaces is important for the reader. Is there something more profound than “one of the two classes is more flexible than the other”? It could be useful for the reader to clarify this aspect, and in particular explain what is preventing the generalization of the result to deeper architectures. I am not sure I fully grasp where the model complexity coming into play in the theoretical development of the paper\n\n2) Propagation of chaos is a technique I am not very familiar with, and I suspect it is not mainstream in the machine learning community. Providing some background on this would be helpful to produce a more self contained paper\n", "summary_of_the_paper": "This paper studies the training of Energy Based Models (EBM). The “standard” technique is maximum likelihood maximization of the observed dataset. While theoretically sound, there are practical difficulties in simulating the MCMC dynamics due to large basins in the energy landscape. For this reason a class of alternative methods, like Score matching, have been explored in the literature. Score matching, however, is deemed to suffer in statistical power.\n\nThis work explores how, leveraging known results about Fenchel dualities, it is possible to connect the two training modalities (Maximum Likelihood and Score based) on a continuum, and proposes a practical algorithm based on such considerations.\nThe authors consider for their theoretical analysis of the expressiveness of the models,  shallow neural networks in the lazy and kernel regime (spaces F_1 and F_2 respectively).\n\nSection 2.1 introduces the two considered spaces, referring to the works of Chizat and Bach. \n\nAn energy based model is defined at the beginning of section 2.2 by means of the Radon Nikodym derivative. Training an EBM model via maximum likelihood corresponds to maximizing eq. (2), that, when considering networks in F_1, can be rewritten as the expression (3). At the end of Section 2.2. score matching is rigorously defined and the relationship between theoretical and empirical implementations (Hyvarinen) is clarified.\n\nSection 3 is the core of the paper. If Assumption 1 holds (growth conditions), then Thm 1 states that eq. (3) is the Fenchel dual of (5). As (5) is a static functional, the authors propose to use results from Chizat and Santambrogio to rewrite it as the dynamical system (eq. (6))  together with the definitions in (7). Importantly, this formulation contains \\alpha, a free parameter >0, that determines relative time-scales over which functions \\gamma and \\nu are updated. System (6) is still “algorithmically” unsolvable. The authors then propose to use the two following approximations: first, the mean field technique known as propagation of chaos, is used to switch from (6)  to the system (8), where gamma is substituted by neural network  parameters and \\nu by and SDE. Then the continuous time system (8) is discretized using Euler-Maruyama. Algorithm 1 is the result of the process.\n\nSection 4 explores how, in the infinite alpha regime, the dynamical system (6) is equivalent to score matching. By means of eq. (11) and proposition (3) the technical analysis is performed. Intuitively, parameters are updated infinitely faster than the samples.\n\nSection 5 explores the implementation of Algorithm 1 on a synthetic dataset. The authors consider two cases (the two teacher neurons aligned to 164 and 78 degrees respectively). Comparison of the primal and dual (alpha<<1, alpha>>1) shows that the dual performs much better than the primal.\n\nThe conclusions stress the theoretical contribution based on Fenchel duality and propose as future work the exploration of convergence rates of (6) and (10) and numerical investigation of the restarting probability p_R. In addition, the authors suggest that “porting” the presented work to deeper network architectures (e.g. to work on realistic dataset, such as images) is straightforward from the practical point of view, and unpractical from the theoretical point of view.\n", "summary_of_the_review": "I think this work has a lot of potential, and the theoretical quest set by the authors is important.\nHowever, the theoretical result per se is not fully exploited, and while the authors state this is on their research agenda, the current paper is weak on the algorithmic (the suggested idea of an annealed p_R should be developed) and empirical sides.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors generalized the Fenchel duality formulation of the maximum likelihood for F1-EBM, which leads to a min-max optimization formulation. Meanwhile, the optimization reveals a new connection between primal-dual MLE with score matching. These contribution is significant and make the paper interesting to the community. 

However, there are several issues need to be addressed. 

- *Experiments*: most of the reviewers concern about the empirical study, which are conducted on synthetic data. The paper will be much stronger if the comparison on real-world dataset, e.g., MNIST and CIFAR10, can be conducted. 

- *Clarification of paper*: I totally understand that due to this is a theoretical-oriented paper, it must be notation and derivation heavy. However, the paper will be much easier for reader, if more discussion is added, e.g., the implementation of the proposed algorithms and more explanation about the comparison between primal-dual algorithm vs. score matching and the experimental results for broader audiences.  


In fact, I personally like the paper very much, which provides a promising solid algorithm for EBM estimation, and the connection to score matching is also novel and different from the current understanding. However, unfortunately, the authors gave up the rebuttal and did not successfully address the concerns from the reviewers. I strongly encourage the authors to revise the draft according to the reviewers' suggestion and resubmit to another venue.
Decision: Reject