Learning Energy-Based Models by Self-Normalising the Likelihood
Authors: Hugo Henri Joseph Senetaire
Paul Jeha
Jes Frellsen
Pierre-Alexandre Mattei
Venue: Accepted by TMLR ()

Abstract: Training an energy-based model (EBM) with maximum likelihood is challenging due to the intractable normalisation constant. Traditional methods rely on expensive Markov chain Monte Carlo (MCMC) sampling to estimate the gradient of logartihm of the normalisation constant. We propose a novel objective called self-normalised log-likelihood (SNL) that introduces a single additional learnable parameter representing the normalisation constant compared to the regular log-likelihood. SNL is a lower bound of the log-likelihood, and its optimum corresponds to both the maximum likelihood estimate of the model parameters and the normalisation constant. We show that the SNL objective is concave in the model parameters for exponential family distributions. Unlike the regular log-likelihood, the SNL can be directly optimised using stochastic gradient techniques by sampling from a crude proposal distribution. We validate the effectiveness of our proposed method on various density estimation and parameter estimation tasks. Our results show that the proposed method, while simpler to implement and tune, outperforms existing techniques for small to moderate dimensions but degrades for high-dimensional problems. We extend this framework to handle EBM for regression and show the usefulness of our method in this setting, as we outperform existing techniques.

Published in Transactions on Machine Learning Research (03/2026)
Learning Energy-Based Models by Self-Normalising the Likelihood
Hugo Henri Joseph Senetaire
hhjs@dtu. dk
Technical University of Denmark
Paul Jeha
pauje dk
Technical Universiby of Denmark
Jes Frellsen"
jefr dk
Technicol University of Denmark
Alexandr Mattei"
Université Côte d' Azur
Inria, CNRS
LJAD, Maasai team
Reviewed on OpenReview: https: // wptp///eee.rv.ee...mme.ee. net/ forum? id= GVaPBqI6ny
Abstract
Training an energy-based model (EBM) with maximum likelihood is challenging due to the intractable normalisation constant. Traditional methods rely on expensive Markov chain Monte Carlo (MCMC) sampling estimate the gradient of logartihm the normalisation constant. We propose novel objective called self-normalised log-likelihood (SNL) that introduces a single additional learnable parameter representing the normalisation constant compared to the regular log- ligliiel. SNL is lower bound of the log- likelihood, and its optimum corresponds to both the maximum likelihood estimate of the model parameters and the normalisation ion constant. We show that the SNL objective is concave in the model parameters for exponential family distributions. Unlike the regular liglliie,,hood, the SNL can be directly optimised using stochastic gradient techniques by sampling from crude proposal distribution. We validate the effectiveness of our proposed method on varions density estimation and parameter estimation tasks. Our results show that the proposed method, while simpler to implement and tune, outperforms existing techniques on small to moderate dimensions but its performance starts to degrade for very high-dimensional problems. We extend this framework to handle EBM for regression and show the usefulness of our method in this sctting as we outperform cxisting tochniques.
1 Introduction
Fasrgy-based models (FBMs) specify probability density over space X' through parameterised energy function Eg X + R. The associated density is then
Eo(ww)
Po (x)
(1)
Za
where Zg = ( dx is called the partition function or the normalising constant. However, Zo is often unknown and intractable, which makes training an EBM through maximum likelihood challenging.
"Equal contribution.

---

Published in Transactions on Machine Learning Research (03/2026)
eNL (0. b)
Initial methods address the challenge with psendo-lkeeihood function, an altered version of the likelihood function that cir-
curvents the need to compute the normalising constant (Besag)
1975, Mardia et al., 2009; Varin et al.) 2011). Alternatively, gradients of the log- log-likelhhodd function can be estimated using
the Boltzmann learning rule (Hinton & Sejnowski 1983) or
approximated using contrastive divergence (Hinton 2002) at
the price of expensive and iffficult-tottune Markov chain Monte
Carlo (MCMC) sampling methods (Dalalyan 2017 Welling &
Teh 2011). To alleviate this difficulty, Du & Mordatch (2019)
proposed to maintain a buffer of samples during training using Langevin MCMC. This work was extended by Du et al. (2021), who considered a Kullback-Leiblee divergence term that was claimed to be negligible by Hinton (2002). Relatedly, Xie et al. (2021) use flow trained alongside the EBM starting point for short-term MCMC sampler, reducing the dependency long chains. In another work, Gao et al.] (2021) proposed train- ing a succession of EBM on data diffused with noise, allowing for both training and sampling the conditional distribution. Ni- ikamp et al. (2019) studied the training of EBM for short-term non-convergent Langevin Markov chains and showed excellent generation, albeit without directly optimising the likelihood. As it is critical to have a good estimate of this gradient, alternative methods consider using a proposal distribution q together with importance sampling (Bengio & Senécal 2003). However, this results in an objective that is an upper bound of the log-likel Additionally, the choice of a proposal is critical, and a poor choice will lead to a loose bound. To tighten it, Geng et al. (2021) train the proposal to mininiss the bound. This results in a min-max objective, similar to that of generative adversarial networks (GANs), which are infamous for their instability in training (Kumar et al.) 2019 Farnia & Ozdaglar 2021
0
The SNL for a Gaussian with un- known mean 0 E R and unit variance. The SNL a function of both 0 and the additional parameter b, estimating the normalising con- stant. The black line corresponds to max- imising b for each given 0, which exactly recovers the log-likelihood. The red star is the maximum log- loeelhooo,,,,, which is also the maximum of ESNL. (0, b), see details in Appendix B
Figure 1:
Another line of work aims al getting rid of the partition function altogether. It notably includes score matching and its variants (Hyvärinen 2005 Vincent 2011). Score matching is a family of objectives that circumvents the normalising constant by matching the Stein score of the data distribution (Stein) 1972) to that of the model. Several variants have subsequently been proposed: implicit score matching trades the Stein score of the data distribution for the Hessian of the model (Hyvärinen) 2005 Kingma & Te Cun 2010 Martens et al., (0122,, while denoising score matching models instead corrupted version of the data, which has tractable density. The latter approach has proven very successful at generating high-dimensional data, such as images and videos (Song et al. 2021 Ho et al., 2022). Another approach that bypasses the normalising constant is to minimise the Stoin discrepancy (Barp et al. 2019 Grathwohl et al. 2020).
An alternative approach more closely related to our work is noise contrastive estimation (NCE), where Gutmann & Hyvärinen (2010) frames the problem as a logistic regression task between the data and a tractable noise distribution. This Icads a consistent estimate of the model paramcters. Additionally, the normalisation constant is learned using an additional parameter (Mnih & Teh 2012). The crucial issue of NCE, and that will not affect our method, is that the objective depends the noise distribution, which is very hard to optimise (Chehab et al. 2022).
1.1 Contributions
Our work is inspired by two papers local likelihood density estimation (Loader 1996 Hjort & Jones 1996), which mention ways of bypassing the normalising constants in their quite specialised context. Our contributions are the following:
We propose new objective, the self-normalised log-likelihood (SNL.) that is amenable to stochastic optimisation and allows for recovering both the maximum likelihood estimate and its normalising constant.
2

---

Published in Transactions on Machine Learning Research (03/2026)
We study theoretical properties of the SNL, in particular its concavity for exponential families and its links with information geometry.
We demonstrate
range of low-dimensee www-dimnniioal tasks, including density estimation and parameter estimation, that SNL, is straightforward to implement and achieves performance comparable to that of more complex approaches for learning energy-based models. We show state-of-the-art results on image regression datasets using an energy-based model.
We derive a surrogate training objective, the SNELBO, for variational autocncoders with an EBM prior, and evaluate it on binary MNIST, CIFAR-10, and CelebA. While this approach improves upon the vanilla VAF baseline, the resulting generations remain below the performance of current. state-of-the-art models.
2
Self-normalising the likelihood
We deal with some data 21,-..;3i X, assumed to be independent. and identically distributed samples from distribution Pdata. Our goal is to fit EBM Pø, defined in Eq. (T, to these data. The standard approach for fitting a probabilistic model is to maximise the likelihood function
{()) = Σ log (1..).
(2)
Unfortunatcly, we will discuss now, maximising such function for EBM is daunting task.
2.1 Why maximum likelihood for EBMs is hard
Let us focus on a single data point x. The log density of our EBM is
log - -Fo(x) log Zo:
(3)
with being the learnable parameters the model. Gradient-based methods are popular approach to train an EBM via maximum likelihood; those methods require the gradient of the log density with respect to the parameters, that is
V logpo(z) (o V e log Ze.
(4)
While automatic differentiation can, usually, easily compute the gradient of the energy (z)) it is not the case for log Za.. However, following the Boltzmann learning rule (Hinton & Sejnowski] 1983). we can express the gradient of the normalising constant as an expected value (sce, c.g., Song & Kingma 2021 for a full derivation):
(5) We can obtain Monte Carlo estimate of this gradient, but this requires sampling from the EBM itself, which leads to the use of MCMC-based methods that often suffer from poor stability and high computational cost. These procedures usually require very long chains to converge to the true distribution pş- For the EBM to be computationally trainable, one noeds to cut short the procedure, and as a result, the obtained samples do not follow exactly pe, meaning that the estimates of Va log Zo are biased. As it is critical to have a good and fast estimate of this gradient, alternative methods consider using proposal distribution q an importance sampling fashion, to yield cheaper estimate:
Ve log Ze = [x
log Zo log / f (ool() dr log
/ -oloe) q(x) ((x) dr 2 Ex, log (Xm)
M Ex(X
(6)
where the last inequality is a consequence of Jensen's 's inequality (Jensen 1905 that we will maximise the likelihood upper bound
1906 In turn, this means
1 M
e Fm()
(0) = Σ
Ea (xx) X....
log
Σ
2 1(0),
(7)
M m=1
g(Xm)
1=1
3

---

Published in Transactions on Machine Learning Research (03/2026)
in lieu of the likelihood. Depending the choice of q and the number of importance samples M, this inequality is potentially very loose, meaning that one would train the model to maximise a biased approximation of the likelihood. Finding a good proposal that allows for fast sampling and correct estimation of its entropy is still a very active research area (Grathwohl et al. 2021 Kumar et al. 2019; Xie et al. 2018).
Usually, this proposal is trained in parallel with the model Eg, which leads to a very unstable advessarial objective (Geng et al. 2021)
2.2 Can make this logarithm disappear?
The looseness of the importance sampling approximation (0) is only due to Jensen's inequality: if the logarithm were replaced by linear function, it would be possible to compute unbiased estimate of the log-likelihood gradients. Our key idea is therefore to linearise the logarithm, using the following simple variational formulation. This will help us bypass the issues mentioned in Section 2.1
Lemma 2.1. For all 0,
log = min (zc-> fee~Axx11 XER
(8)
The proof of this lemma is elementary and provided in Appendix A.1 This result is often used as an illustration of variational representations in variational inference tutorials (see, e.g., Jordan et al.) 1999.
Section 4.1; Ormerod Wand 2010, Section 3), but we are not aware of being used in a context similar to ours. Applying Lemma 2.1 to Eq. (3) give us, for any x E X.
Pog)o(x)) = Fo(()) log 7.0 -Fo(x) - min
c (e 70 b b-b
bER
(9)
(-e- - - b +
(-Eo(x)- - eb-e- +1)
= Eo(r + max BER
max BER
Using Eq.
we define new objective named the self-normalised log-likelihood (SNL) LNN, that is a function of the original parameter of the EBM 0 and a single additional parameter E R:
1 Σ (SNL (b,0)
Ee(zz) - b- e Zo +1.
(10)
When maximised w.r.t. b, we can recover the exact log-likelihood of given model pa and maximising both 0 and b leads to the maximum log-likelihood estimate, as formalised below.
Theorem 2.1. For any given 0, when the SNL is marimissd with respect to b, we have access to the exacl log-likelihood of the model:
max (S,I, = E(())
(11)
BE3
Moreover, the optimum, b the normalisation constant::
arg max (SNL(0, b) = log Zo. bER
(12)
Finally, there is a one-to-one correspondence between the local optima of the SNL and the log-likelihood. The proof is available in Appendix A.2 and is a simple application of the variational formulation of the logarithm. The important consequence of this result is that marimising the SNL w.r.t. 0 and b will recover both the marimun log-likelihood estimate and its normalising constant. This ability of our objective to learn both the model and its normaliser motivates the name self-normalised log-likelihood. We chose to call the extra parameter because, when Eg is modelled as neural network, b can simply be understood as the bias of its last layer. In Appendix E.1 we propose another interpretation of SNL that derives directly from the Donsker- Varadhan variational representation of the KL distribution.
Another direct consequence of Eq. g is that, for any 0 and b, SNL is lower bound of the log-likel Using the importance-sanpling upper bound, this will lead to useful "sandwichings" of the log-iiklihood:
(sNI(0,0) (0, E (()) (0)
(13)
4

---

Published in Transactions on Machine Learning Research (03/2026)
2.3 Why maximising the SNL is easier
Why is the SNL more tractable than the standard log-likelihood? After all, the SNL also involves the intractable normalising constant. The key difference is that, since it depends linearly on it, it is now possible to obtain unbiased estimates of the SNL gradients. Indeed, using a proposal q gives us estimates the gradient of Zg with importance sampling. Using
/ Eolx)
T-- Ex())
Zo =
(())) q())dr = Exaq
(14)
(())
allows to get unbiased estimates of the SNL gradients w.r.t. 0 and b. More precisely, for batch of size NB and a number of samples M, we use the following estimate of the gradient w.r.t. 0:
VoEo (X)e(X))-Bull (x(
(0,sN)(0,b)
V VoEa(r;) +e -x-0x
((X)
(15)
n
VoLi( te 1
Σ
Vo Fio Bo(am) (((mm)
BB
M
Similarly, we can compute unbiased estimates of the gradients w.r.t. b:
VolsNL(0,b) -1 e Exag
(x)
- 1 M
Σ
Eo(um)
(16)
((X)
((
The theory of stochastic optimisation (sce, o.g., Bottou et al. 2018 then ensures that SGD-like algorithms, when applied to SNL, will converge to the maximum likelihood estimate and its normalising constant. In practice, we popular algorithms like Adam (Kingnaa & Ba 2015) to train 0 and b jointly. Some specialised algorithms could also be used. For instance, Bietti & Mairal (2017) call optimisation problems similar to ours "infinite datasets with finite sum structure" (in our case, the infinite dataset is samples from the proposal, and the finite sum corresponds to the actual data), and propose an algorithm fit for this purpose. The full algorithm for training an energy-based model with SNL is available in Algorithm [
Are the gradients of l and fSNL related? If we rewrite this gradient in the same fashion as Eq. (4 we can express the gradient the SNT, with the gradient of the log-likelihood:
VolsNI.
VoEo(x) - bbtlog Zo V log Zo Volo + Volog Zo ebbtloggay
(17)
When b equals the normalisation constant log Zo. we obtain an unbiased estimator of the true log- big-likelhhood gradient.
2.4 Practicalities when using SNL
For EBMs to be well-posed, it is required that the normalisation constant exists, i.e., that, fe Fe(2) dx < co. To
that end, following Grathwohl et al. (2020) and similarly to exponential tilting (Siegmund 1976), multiplying the un-normalised probability by a density d ensures the existence of the normalisation constant. The distribution becomes po( xx eEoo())((x).
We call d the base density or the base distribution. In the case where the proposal g is equal to the base distribution, the SNL estimates and the gradient estimates simplify:
M
VoZa
VoEo (e Eo(zm),
(18)
M
m==
Furthermore, we initialise b by estimating log Za with importance sampling using the proposal g the beginning of the training procedure. This practice allows us to get gradient estimates SNL somewhat close to the true gradient log-likelihood.
5

---

Published in Transactions on Machine Learning Research (03/2026)
2.5 Related works
Objectives similar to SNL have been proposed in the past. In particular, in the context of local likelihood density cstimation, Loader 1996) and Hjort & Jones (1996) handled intractable normalising constants in a similar fashion to ours. Arbel et al. (2020) leveraged a similar approach to estimate the normalisation constant to train hybrids of generative adversarial nets and EBMs. Neither of these works used importance sampling. Pihllaja et al. (2010) and Gutmann & Hirayama (2011) proposed families of generalisations of NCE which contain an objective similar to SNL as special case. In these generalisations of NCE, the noise distribution plays a similar role to our proposal, but the obtained estimates in general differ from maximum likelihood. The novelty of SNL lies in the fact that it allows for performing exact maximum likelihood optimisation (regardless of the choice of proposal) for an EBM using stochastic optimisation together with importance sampling.
The SNT. objective is related to the Donsker- Varadhan representation of the Kullback-Leibler diver- gence (Donsker& & Varadhan 1975), variational formulation that has inspired several related approaches (Ar- bel et al. 2020 Belghazi et al. 2018 Glaser et al.) 2021 SNL arises by evaluating the Varsdhan dual
at specific risatint h(x)) -- [z) - b, where b learns the log-partition function (see Appendix E.
Two closely related methods also build on this variational foundation but differ in key ways. The Generalized Energy Based Model (GEBM) (Arbel et al. 2020) uses the same parametrisation as SNL, but treats the base measure B as a learnable implicit model (e.g., a GAN generator), jointly optimising both the energy and the base via alternating updates. A key distinction that in SNL,, the base mcasure d(x:) defines the probabilistic model while the proposal ((x)) used for importance sampling can be chosen independently without affecting the optimum; in GEBM, no such separation exists, samples for estimation learned base B. KALE Flow (Glaser et al. 2021) takes different approach, optimising over restricted function class H with regularisation to define a surrogate divergence for gradient flows rather than maximum likelihood estimation. We discuss these connections and differences in detail in Appendices E.3 and E.4
drawn directly from the
3
Some theoretical properties of SNL
3.1 Concavity of SNL for exponential families
Tt is a well-known fact that the log-likelihood of exponential families is concave because the particular form of the gradient of the normaliser. We provide a proof in Appendix [A.3 [A.3 for completeness. The self-normalised log-likelihood preserves this property with the exponential family: the SNL is even jointly concave in both parameters.
Theorem 3.1. If (po)s is a canonical exponentiol family, then (0, b) is jointly concave.
The proof is available in Appendix A.4 and follows directly from the convexity of the exponential. This means that the many theoretical rosults stochastic optimisation for convex functions could be leveraged to prove convergence guarantees of SNL (see, e.g., Bottou et al. 2018).
3.2 An information-theorette interpretation
Maximum likelihood has the following classical samples goes to infinity, maximising the likelihood is equivalent to minimising the Kullback-Leibler divergence between pe and the true data distribution Paata (sce, e.g., White 1982 A similar rationale also exists for SNT. and involves a generalisation of the Kullback-Leibler divergence to un-normalised finite measures. This generalisation exists also in the more general context of f-divergences, as detailed for instance by Amari & Nagaoka [2000] Section 3.6) or SStummer & Vajdal (2010). It reduces the usual definition when Si and f2 are probability densities and shares many of the merits of the usual Kullback-Leible divergence (see Appendix D for more details).
interpretation: when the number of
Standard maximum likelihood is asymptotically equivalent to minimising (L(pdatallpo) As we detail in A Appendix D this turns out to be equivalent to minimising the generalised divergence between Pdata and all un un-normalised models proportional to e -Ep;
KL lpp) min (L Ice (Eo)
(19)
c>0

---

Published in Transactions on Machine Learning Research (03/2026)
This new divergence is related to the SNL in the same way that the standard Kullback-Leible divergence is related to the likelihood. Indeed, for any c e> O,
KL(Puatalle (Fo) = / log
Pdaca(2) ce Eo(x)
Pdaca (2) + cZg -
(20)
/
(x()))
Pdata (r) cogc+ + cZp 1+
log(Pdata (F)) (x)dz.
(21)
does not depend on B nor c
The first integral, that depends on 0, is intractable, but may be estimated if we have access to an ii.d. dataset 21, 21,..., Eni leading to the estimate
n Eo(xs) logg e + cZo - 11 + / |log (og(panta (1))) (2) da
(T lIee -ee=
(22)
11.
= loge) + / log(paasa(()))
dæ,
(23)
which means that minimising the SNL will asymptotically resemble minimising the generalised Kullback- Leibler divergence. In the context of local likelihood density estimation, Hjort & Jones (1996) also derived
similar connections with the generalised Kullback-Leii Leibler divergence. More recently, Bach (2022) applied
the same variational representation of the logarithm to the generalised Kullbake-Leibler, context very different from ours.
4
Extending SNL beyond basic density estimation
4.1 Truncated densities
Truncated densities are probability density functions defined truncated domains. They retain the parametric form as their non-truncated counterparts, differing only by normalising constant. However, because this normalising constant is often difficult or impossible to compute analytically, applying Maximum Likelihood Estimation to truncated density models becomes challenging. Even for a simple Gaussian distribution with more than two dimensions, estimating such parameters is not straightforward. We restrict the study to the truncated density of the following shape:
pb(x)) ((x)
(24)
Z
where (x) a standard density, t: x (1,0) the known truncation function and Z/ the unknown normalisation constant. The corresponding energy function for an EBM then
( log))(<)) tf 11
(x(2)
(25)
00
else.
4.2
Self- celfnoormalistion in the regression setting
We consider the supervised regression problem where given dataset of pairs of inputs and targets (I,w) E X x y' where the target space ) is continuous. We want to estimate the conditional distribution Pdata (v/z) using an EBM:
po(u|æ) e (Eo(z.))
(26)
Za,
where Zo,r = f-- dy. The main difference with the previous density estimation setup is that the normalisation constant Zø also depends on the input value x.
Because the normaliser now also depends on x, we introduce new family of functions by whose role is to estimate the normalisation constant Ze Similarly the density estimation case, we define the self-normalised log-likelihood as
ESNL (0, --- Σ
( Eo(o,, v.) - bo(()) Zo. ea bbatee) + +1).
(27)
1=1
7

---

Published in Transactions on Machine Learning Research (03/2026)
Provided the family by is expressive enough, this SNL for regression enjoys the same properties its unsupervised counterpart. We can retrieve the maximum likelihood estimate when maximising the SNT. in both 0 and 6. Moreover, at the optimum, for any I E X, bz(x) is the normalisation constant log Ze. The SNL for regression also a lower bound of the true conditional log-likelihod. Following the reasoning of Section 2.3 we propose to train an EBM model for regression using the SNL. To that end, we consider a proposal dr that depends on both x and y and paramcterised by th. For instance, Gustafsson al. (2020) use mixture density network (MDN, Bishop 1994) proposal. In the work of Gustalsson al. (2020), the EBM is trained jointly with the MDN. The MDN maximisation objective is average combination between the negative Kullback-Leibler divergence between the po and Go.
In concurrent work, Sander et al. (2025) introduced the SNL loss of Eq. 27
for regression in the context
of multi-label classification and label ranking. They extend Theorem 2.1 to the same neural werw
parametrisation of the normalisation constant by and study the connection to broader Fenchel- Young losses and the gencralisation capacity of the objective.
4.3 nelf-normalised evidence lower bound
The SNL approach allows training a variational auto-encoder (VAE, Kingma & Welling 2014 with an energy-basod prior using approximate inference. Both Pang al. (2020) and ? trained an EBM as a prior the latent space for noisy sampler, but required MCMC to sample from the posterior and the prior during training. We introduce the nelf-normalised evidence lower bound (SNELBO), a surrogate ELBO objective that leverages the self-normalise log- likelihood to allow for straightforward training,
Formally, we consider a VAE with a prior po(z) defined by an EBM composed of an energy function Eg
--Bad(s) where
parameterised by a ncural network and an associated base distribution d(z), i.c., po(=) Zo Z fe S-efeeeeiee (e=Bo())(=)dz= The generative model is the same as in VAE, and output density po(x|2) is paramcterised by neural network (2)... Since the likelihood is intractable, we posit conditional variational distribut ion (=|xr) to approximale the posterior of the model, similarly to the original VAE. Using Lemma 2.1 we can obtain the SNELBO:
log d(z)
CSNI. (O, 10,0,71 6) = (xl() [log (o + (sla))
I-y (z(2r)
+ E 1 (zla) (-Eo(2) - -]]= [e --
+ (28)
We note that the SNELBO is a lower bound on the log- log-likelihood, E(0, ()) and a lower bound on the regular ELBO, L, that is tight for optimal 6, i.e., E(0,0) > L(O, v) 2 (SNL o, 7,6) and ((0,0,7) = maxteR CSNL (0, Φ, y, b). See Appendix G for derivation details. This surrogate objective can be interpreted as the combination of the ELBO from a VAE whose prior is the base distribution d(z) with regularisation term from the EBM. As such, the EBM can be added easily top of any VAE model.
5 Experiments
In this section, we employ SNL for different applications. While SNL does not achieve state-of-the-art performance in every case, it provides an easy-to-use, tut-of-the-box solution for problems where the normalising constant is unavailable. Although extending SNL to high-dimensional, arbitrary density estimation remains challenging, it works seamlessly for directional and truncated distributions, which are otherwise nontrivial cases. Moreover, we obtain state-of- taate-of-thh-art results with SNL applied energy-based models (EBMs) for image regression.
5.1 Density estimation
5.1.1 Density estimation for directional distributions
As presented Theorem 3.1 the SNL objective is concave for exponential families. This makes the objective very attractive for exponential families with unknown normalisation. This includes many directional distributions, such as the multivariate von Miscs (Mardia et al. 2008) for which no tractable formulas exist. The multivariate von Mises distribution has a density of the form
McM(x)= Zo Zo.
exp (N. (eFc(r, 0) + s(x, 0)2 As(x,)))
(29)
8

---

Published in Transactions on Machine Learning Research (03/2026)
40
3.20
15
2
Grien Geup2
Grip3 Gh-44
Croup
Goup3
Gesp4
Error on the estimated parameters obtained using different likelihood methods (approximate maximum likelihood, MT.), pseudo likelihood (PT), self-normalising likelihood (SNL. and score matching (SM, Mardia al.) 2008). The score and uncertainty for ML, PL and MM are directly reported from Mardia ct al. (2008) while report average scores of SNL and SM over five different generated datasets and runs. The exact value of the estimated parameters is given in Table 17 of the appendix.
Figure 2:
with A. an unknown normalisation constant, the localisation parameter, A the correlation param- eter and k the concentration parameters. These parameters satisfy - < 0 I n, - 1 I x and K2 z O. The matrix A real valiod and symmetric with (A)i Aij = dzt for + j and
diagonal elements ds 0. We defined c(x,0)² (cos (x 01), cos (x 02) ...., cos (x - 0p) and
s(x, 0y* sin (r 01) sin (a - 02), ........... sin (x-0p)) sin We evaluate the performance of our SNL estimator using datasets sampled with four different sets of parameters of the multivariate von Mises distribution, following the setup of Mardia et al. (2008) Following their experimental set-up, we fix the localisation parameters to their true value and only estimate the concentration and correlation parameters. Each set of parameters was designed to explore the possible combinations with high/low concentration i and high/low correlation Xij- In Figure 2 we compare our estimator to score matching (SM) by Mardia et al. (2016), pscudo-likclihood and approximate maximum likelihood with numerical integration by Mardia et al. (2008). SNL outperforms the other estimators in most. cases, except in the low correlation case where the variance is high.
5.1.2 Density estimation for truncated distributions
We evaluate the performance SNL. on a truncated distribution using a simple test without model misspeci- fication from Melchior & Goulding (2018). We draw samples from a Gaussian Mixture Model with K = clusters truncated by a box and a circle with known boundaries. As opposed to the original implementation by Melchior Goulding (2018), we do not add noise to the sample and simply discard samples outside these boundaries to create the truncated distribution. The details of our parameterisation provided in Appendix [.2]
We evaluate the log-likelihood of the resulting model (without truncation) on the non-truncated test datasel. in Fig. 3 We further evaluate the quality of the estimation by comparing the parameters of the estimated mixture to the true parameters
Fig. S Although the performance in likelihood does not match the specialises GMMis implementation of Melchior Goulding (2018), the simplicity of our approach suggests that SNL constitutes a promising and flexible alternative for truncated density estimation.
5.1.3 Density estimation for tabular data
In this section, evaluate the capacity of SNL to train an EBM for density estimation. We consider both artificial datasct and real datasct from UCL. In both cases, we compare our model with an EBM trained in the same condition but with noise contrastive estimation (NCE, Gulmann & Hyvärinen 2010).
We evaluate the performances of EBMs for density estimation, trained with SNL on four different, two- dimensional, generated datascts. We compare our model with an EBM trained the same condition but with noise contrastive estimation. Both setup also leverages base distribution that equals the proposal distribution. Qualitatively in Fig. G observe that the two models perform par, except for the four
9

---

Published in Transactions on Machine Learning Research (03/2026)
Tue Distribution
GMYis logc = -4077
SNL log 4.172
log c 5.994
log = - 4.856
Figure Different methods are used to estimate the parameters of a truncated mixture of Gaussians. Each method is fod with truncated data (the blue points). The log-likelihodd evaluated on complete test set (i.e. without truncation) in order to verify the quality of the estimation. From left. to right, we show the true density of the GMM, the model obtained with standard Expectation maximisation algorithm, the model obtained with EM using imputation (GMMis Melchior & Goulding 2018)) and our model trained using gradient descent and SNL.
Funnel
Pinwheel
Checkerboard
Four Circkes
Objective Base Dist
fis
ESNL
E1
ESNL
fs
ESNL.
ÉSNT
NCE
Nio. 1)
-2,040 :20 mm -2,044
100.204)
-1.004
-1.020
(10.064)
-1.947
120.000)
-1.964
(20.002)
-2.117
(an.00)
-2.120 (10.000)
SNI.
Nio,1) 1)
-1.811 (e:
-1.031
(ze ony
-1.035
-1.902
-1.905
-1.914
(d mony
-1.918
(e o09
NCE
None
-1.804
-1.896
-1.063
may -1.069
m2)))))
-1.997
enn;
-2.025
nos)
-2.231
-2.232 axo)
SNL
Noue
-2,006 ta0
-2.066
-10072 (20.003)
- -1.086 (20000 -1.966
-1.909
-1.971 (anar)))
-1.973
Table 1: Evaluation of the performance of EBMs trained with NCE or SNL. objective with or without a base distribution. We generate each dataset five times and run each set of parameters once on each. We report the mean and standard deviation of the estimated log-likelihodd and the self-normalised likelihood fSNL- Highest best.
circles dataset, where SNL dominates. We explore the impact of the base distribution in this setting. We show our results in Table D According to those results, SNL-trained EBMs with a base distribution perform better than all the NCE settings across all datasets except Pinwheel. Using a base distribution always improves the performance with SNL, while it varies with NCE.
We also evaluate SNI-trained EBMs on UCT datasets order to assess the impact of incrcasing dimensionality. We use a simple Gaussian proposal with full covariance. The results, reported in Table show that EBM-SNL is competitive
dimer-diieasoonal tabular datasets, while its performance degrades compared to strong normalizing-flow baselines on Hepmass (d = 21). Despite this drop in performance at higher dimensions, the method remains appcaling in low-dimensional settings due to its simplicity, relying only on fully connected neural network for the EBM and a simple Gaussian proposal.
5.2 EBMs for regression
Following Gustafsson et al. 2022 we study and compare our training method on two one-dimensional regression tasks (seeFigg 5 in Appendix H and four image regression datasets. We parameterise our model
with the same architecture as Gustafsson et al. (2022) where the output feature extractor, hg, feeds both the proposal go (.)h2) and a head neural net work for the EBM (see Figure in Gustafsson et al. (022 for
more details). When used as a proposal, the weights of the fcature extractor are frozen.
EBM SNL.
Ganseian
MADE
MADE MoG
NVP
Real NVP (10)
MAF (5)
MAF (10)
MAF (5)
Power (d 10 6)
[0.28. 0.41]
-7.74
-3.08
0.10
-0.02
0.17
0.14
0.21
0.30
Gas (d = 8)
[5.73. 7.74]
-3.58
3.56
8-17
4.78
8.33
9.07
10.08
9.59
Hepmase = 21)
[-19.22 19.20
-27.3
-20.98
-15.15
-1877
-17.70
-17.73
-17.39
Table 2:
For FBM-SNT. the upper bound corresponds to Ers and the lower bound to ESNL- Both are
computed using 20000 samples from the test set.
10

---

Published in Transactions on Machine Learning Research (03/2026)
NCE
-30
3.00
3.25
-34
26
-3.75
4..00
16
32
128
Nb Samples
No Samplies
(a) Cell Count
(b) UTK Faces
Figure This graphs depict how the number of samples M in the proposal affects the performance of the EBM for regression. Each model is trained the Cell Count and UTKFaces dataset with Gaussian proposal.
In our experiments, consider three different proposals:
mixture density network proposal whose parameters are given by small fully connected neural network.
fixed multivariate Gaussian (M, Σ) whose parameters are estimated before training with the training dataset and fixed during training.
A fixed uniform distribution U/ that is defined by leveraging the knowledge from the dataset and fixed during training.
All models are evaluated using an estimate of the log-likelihood with M 20,000 samples vf from a multivariate Gaussian whose parameters are estimated before training:
1 NV Σ
M
-Eo(x,, v4) bx(x)) log Σ E (m)))
Lis
(30)
Since NCE normalises the EBM at the optimum (Mnih & Teh 2012). we also provide the (SNI. (i.e. lower bound estimate of the lieeliie) for each set of parameters using the same proposal Eıs with 20,000 samples. If ESNL. is close to f1s, this means that the lower-bound is tight and by approximates correctly log Zo (or if no by is used the network is self-normalised and Zg = 1).
5.2.1 1D regression datasets
We consider here the two artificial datasets for 1D regression with multimodal distributions p(y|z) (see Fig. 5). We provide description of the neural network architecture in Appendix I.4.1 On both datasets, the SNL always outperformed its NCE counterparts with respect the estimated upper bound lis, seen in Table 3
Moreover, the (SNI. of the NCE is loose compared to the fSNI.- This is due to poor estimation of the normalisation constant with NCE. We provide additional results in Table 20 Using a base distribution to ensure the existence the normalisation constant Ze cither improves or gives similar results with the SNL objective but systematically damages the results when minimising the NCE loss. As mentioned by Mnih & Teh (2012), with both objectives, explicitly modelling by does not provide a better estimation of the network. The normalisation implicitly learned with Eg.
5.2.2 Image regression datasets
We train an NCE-EBM setup and an SNL-EBM setup on an image regression task. We Irain on four different. datasets, steering angle, cell count, UTKFaces and BIWI and follow the same setup as Gustafsson et al.
(2022). Similarly the 1D regression datascts, SNI-trained EBM always outperforms its NCE counterparts Table 4. When using NCE, the normalisation constant off resulting in a loose lower bound of the likelihood whereas SNL usually provides better approximation. Tn Fig. observe that method improves with
11

---

Published in Transactions on Machine Learning Research (03/2026)
Regression Dataset 1
Regression Dataset 2
Objective
Proposal g
Eis
ESNI.
ESNI.
NCE
N(H, Σ)
-0.030 (40.278)
-0.718
-2.592
ree 2143
-3.559 (2:3394)
NCE
MDN K2
-0.611
(de. 154:
-1.492
(de wos:
-2.451
(de (ge)
-2.634 (cn-ssa)
SNL. Nig, E)
0.164
0.033
-1.813
(ee 1003
-1.836
(40 109)
SNT.
MDN K2
0.255
(an 017)
0.251
-2.099
(ee 200)
-2.170
(2 aoa)
Table 3:
Evaluation of regression EBMs on the 1D toy regression problems with two different objectives and two different proposals. Each model is trained for five runs and we report the mean and standard deviation of the estimated bigllikelhoodd lis and the self normalized log- oog-liklihood LSNL. Using the SNL as objective clearly outperforms the NCE.
Steering Angle
Coll Count
U
Objective Proposal
FUNL
fis
FSNL
Eys
Aps
FEND
NCE
X)
-3.649 (21
UNNORMALIZER
-3.367
1203
-9.675
-3.147
-8222
(aam
-11.02
UNNORMALIZED
NCE
MDN-8
-1.001 (1005))
UNNORMALIZED
-3.861
(100:: UNNORMALIED
-4.123
-5.170
(Loaat))
-11.998
UNNORMALIZED
SNL
N),
-2.665 ciiam)
-3.973
(13.45
-2.701
(10as; -2.725
(10.55
-2.966
:11.00m
-2.991
(10.sco)
-10.86
-11.05 (11)
SNL
Uniform
-1.402 (20000))
-.423
2.0004
120.0 -2.620
:20.007
-2.927
(23.000
-2.965
(200015)
-10.44
120 108:
-10.51
(2) 2039
SNL
MDN-8 -1.673 (10.013)
-16692 (20000)) -2.801 (10071: -2.811
(10007s:
-2.921
(10.025)
-2.913 (LO.SCO)
-10.01
-10.04 110.0011
Table 4: Evaluation of EBMs for regression on image regression datasets with two different objectives and different proposals. Each model is trained for five runs, and we report the mean and standard deviation of the estimated log-likelihood (Eıs) and estimated self-normalised log-likelihood (LSNL). When the proposal is MDN, the proposal is learned jointly with the model following Gustafsson el al. (2022).
the number of samples but stagnates after M = 64 samples. On the other hand, NCE seems to improve with the sample size but in a less compelling fashion. We provide additional results in Table 21
5.3 VAE with latent prior EBM
We train VAE with EBM prior on binary MNIST using SNELBO, as outlined in Section 4.3 We paramcterise the output distribution with a Bernoulli distribution with parameters from a neural network Jo: Lee. po(a]2) = (2x) and the approximate posterior with a Gaussian whose parameters are given by a neural network q- (z|x). We cither train from scratch the VAE with EBM prior (VAE-EBM) or we only train the prior of pre-trained VAE with standard Gaussian prior (VAF-EBM Post-hoc). We compare to a standard VAE with a Gaussian prior (VAE) and a VAE with a Mixture of Gaussian Prior (VAE-MoG) and 10 mixtures. All VAEs are trained with a latent space of size 16. In Table 5 we show that training VAF-EBM with latent EBM provides better SNELBO.
VAE
-89.10
VAE-MoG VAE-EBM Post-Hoc
-88.73
-88.11
VAE-EBM
-87.09
Table 5: ELBO/SNELBO for VAEs with different priors.
We report the FID scores of generated samples for standard VAE, a VAE with a learned EBM prior, and a latent short-term MCMC approach Pang et al. (2020) in Table 19 While the learned EBM prior yields only marginal improvement over the vanilla VAE, the latent short-term MCMC method produces substantially higher-quality samples. This improvement, however, comes at the cost of losing direct access to ELBO SNELBO estimates.
6 Conclusion
We proposed a new objective to train energy-based models (EBMs) called relf-normalising log- likeliihoood (SNL). By maximising SNL with respect to the parameters of the EBM and an additional single parameter b, can recover both the maximum likelihood estimate and the normalising constant at optimality. We conducted an extensive experimental study on parameler estimation for directional and truncated distributions, low- dimensional datasets for density estimation, complex regression problems and training VAEs with EBM prior. These experiments illustrate that SNL can casily handle intractable normalising constants many different situations. However, in high-dimensional settings, the variance of the importance sampling estimates used to approximate the partition function and its gradients grows rapidly, which limits the practical scalability of
12

---

Published in Transactions on Machine Learning Research (03/2026)
the method. Reducing this variance, for instance through more expressive proposals variance reduction techniques, will be the subject of future work.
References
S Amari and H Nagaoka. Methods of informalion geomelry, volune 191. American Mathematical Soc., 2000.
M Arbel, L Zhou, and Gretton. Generalized energy based models. In International Conference on Learning Representations, 2020.
F Bach. Sun-of-squares relaxations for information theory and variational inference. arXiv preprinl WwXiv22206.1325. 2022.
A Barp, F.X Briol, A Duncan, M Girolami, and L Mackey. Minimum Stein discrepancy estimators. Advances Neural Information Processing Systems, 32, 2019.
M.I Belghazi, A Baratin, S Rajeshwar, S Ozair, Y Bengio, A Courville, and D Hjelm. Mutual information neural estimation. In International conference on machine learning, pp- 531-540. PMLR, 2018.
Y Bengio and J.S Senécal. Quick training of probabilistic neural nets by importance sampling, In International Workshop on Artificial Intelligence and Statistics, pp. 17-24 PMLR, 2003.
J Besag. Statistical analysis of non-lattice data. Journal of the Royal Statistical Society: Series D (The Statistician), 24(3):179 195, 1975.
D Best and N.I Fisher. Efficient simulation of the von mises distribution. Journal of the Royal Statistical Society: Series C (Applied Statisties), 28(2):15--177, 1979.
A Bietti and Mairal. Stochastic optimization with variance reduction for infinite datasets with finite sum structure. Advances in Neural Information Processing Systems, 2017.
C.M Bishop. Mixture density networks. Neurol Computing Research Group report: NCRG/94/004, 1994.
L Botton, F.E Curtis, and J Nocedal. Optimization methods for large-scale machine learning. STAM review, 60(2):223 311, 2018.
O Chehab, A Gramfort, and A Hyvärinen. The optimal noise in noise-contrastive learning is not what you think. In Uncertainty in Artificial Intelligence, pp. 307-316. PMLR, 2022.
A Dalalyan. Further and stronger analogy between sampling and optinization: Langevin monte carlo and gradient descent. In Conference on Learning Theory, pp. 678 689. PMLR, 2017.
A.P Dempster, N.M Laird, and D.B Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical society: series B (methodologicaal) 39(1):1 22, 1977.
M.D Donsker and S.S Varadhan. Asymptotic evaluation of certain markov process expectations for large time, Communications on Pure and Applied Mathematics, 28(2):279 -011 1975.
Y Du and I Mordatch. Implicit generation and modeling with energy based models. Advances in Neural Information Processing Systems, 2019.
Y Du, S Li, J Tenenbaum, and I Mordatch. Improved contrastive divergence training of enorgy-based models. In International Conference on Machine Learning, 2021.
F Farnia and A Ozdaglar. Train simultaneously, generalize better: Stability of gradient-based minimax learners. In International Conference on Machine Learning, pp. 3174-3185. PMLR, 2021.
R Gao, Y Song, B Poole, Y.N Wu, and D.P Kingma. Learning energy-based models by diffusion recovery likelihood. In International Conference on Learning Representations, 2021.
C Geng, J Wang, Z Gao, J Frellsen, and Hauberg. Bounds all around: training energy-based models with bidirectional bounds. Advances in Neural Information Processing Systems, 2021.
13

---

Published in Transactions on Machine Learning Research (03/2026)
P Glaser, M Arbel, and A Gretton. Kale flow: A relaxed kl gradient flow for probabilities with disjoint support. Advances in Neural Information Processing Systems, 34:8018-8000 2021.
W Grathwohl, K.C Wang, J.H Jacobsen, Duvenaud, and R Zemel. Learning the stein discrepancy for training and evaluating energy-based models without sampling. In International Conference on Machine Learning, 2020.
W.S Grathwohl, J.J Kelly, M Hashemi, M Norouzi, K Swersky, and D Duvenaud. No {memc} for me: Amortized sampling for fast and stable training of energy-based models. In International Conference on Learning Representations, 2021.
F.K Gustafsson, M Danelljan, R Timofte, and T.B Schön. How to train your energy-based model for regression. In British Machine Vision Conference (BMVC), 2020.
F.K. Gustafsson, M Danelljan, and T.B Schön. Learning proposals for practical energy-based regression. In
International Conference on Artificial Intelligence and Statistics, pp. 4685-4704. PMLR, 2022.
M Gutmann and J.I Hirayama. Bregman divergence as general framework to estimate unnormalized statistical models. In Proceedings of the Twenly-Sevee Seventh Conference on Uncertainty in Artificial Intelligence, pp- 283 290, 2011.
M Gulmann and A Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 297-304. PMLR, 2010.
K He, X Zhang, S Ren, and J Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pallern recognilion, pp. 770 778, 2016.
G.E Hinton. Training products of experts by minimizing contrastive divergence. Newral computation, 14(8): 1771-1800, 2002.
G.E Hinton and T.J Sejnowski. Optimal perceptual inference. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, volume 448, 1983.
N.L Hjort and M.C Jones. Locally parametric nonparametric density estimation. The Annals of Statistics, pp. 1619-1647, 1996.
Ho, T Salimans, A Gritsenko, W Chan, M Norouzi, and D.J Fleet. Video diffusion models. arXiv:2204.03450 2022.
A Hyvärinen. Estimation of non-normalized statistical models by score matching, Journal of Machine Learning Research, 6:695-709, 2005.
Jensen. Om konvekse funktioner og ulighoder imellem middelvarrdie. Nyt tidsskrift for matematik, 16: 49 68, 1905. ISSN 09093524. URL http:://w::: jstor. strg//ttable224223322
J Jensen. Sur les fonctions convexes et les inégalités entre les valeurs moyennes. Acta mathematica, 30(1): 175-193, 1906.
M.I Jordan, 7. Ghahramani, T.S Jaakkola, and L.K Saul. An introduction to variational methods for graphical models. Machine learning. 37:183 233, 1999.
D.P. Kingma and J Ba. Adam: method for stochastic optimization. In International conference on learning representations, 2015.
D.P Kingma and M Welling. Auto-cncoding variational Bayes. International Conference on Learning Representations, 2014.
D.P Kingma and Y Le Cun. Regularized estimation of image statistics by score matching. Aduances in Neural Information Processing Systems, 2010.
14

---

Published in Transactions on Machine Learning Research (03/2026)
R. Kumar, S Ozair, A Goyal, A Courville, and Y Bengio. Maximum entropy generators for energy-based models. Xiviv preprint 1xX:11100 000008808, 2019.
E.L. Lehmann and J.P Romano. Testing statistical bypotheses (4th edition). Springer, 2022.
C.R. Loader. Local likelihood density estimation. The Annals of Stotistics, 24(4):1602-1612 1618, 1996.
K.V Mardia, J.T Kent, and A.K Laha. Score matching estimators for directional distributions. arx preprinl 0rXiv:1604.084700 2016.
K.V Mardia, G Hughes, C.C Taylor, and H Singh. A multivariate mises distribution with applications to bioinformatics. Canadian Journal of Statistics,
2008.
V Mardia, J.T Kent, G Hughes, and C.C Taylor. Maximum likelihood estimation using composite likelihoods for closed exponential familics. Biometrika, 96(4):975-982 10 2009.
J Martens, I Sutskever, and K Swersky. Estimating the hessian by back- proo--popaggting curvature. In International Conference on International Conference on Machine Learning, 2012.
P Melchior and A.D Goulding. Filling the gaps: Gaussian mixture models from noisy, truncated or incomplete samples. Astronomy and computing, 25:183 194, 2018.
A Mmih and Y Teh. A fast and simple algorithm for training neural probabilistic language models. In International Conference on Machine Learning, 2012.
E Nijkamp, M Hill, S.C Zhu, and Y.N Wu. Learning non-convergent non-persistent short-run meme toward energy-based model. Advances in Neural Information Processing Systems, 2019.
J.T Ormerod and M.P Wand. Explaining variational approximations. The American Statistician, 64(2): 440-153,40 2010.
B Pang, T Han, E Nijkamp, S.C Zhu, and Y.N Wu. Learning latent space energy-based prior model. Advances in Neural Information Processing Systems, 2020.
A Paszke, S Gross, F Massa, Lerer, J Bradbury, G Chaman, T Killeen, 7 Lin, N Gimelshein, L Antiga, A Desmaison, Kopf, E Yang, Z DeVito, M Raison, A Tejani, S Chilamkurthy, B Steiner, L Fang, J Bai, and S Chintala. Pytorch: An imperative style, high-performanee deep learning library. Advances Neural Information Processing Systems, 2019.
M Pihlaja, M Gutmann, and A Hyvärinen. A family of computationally efficient and simple estimators for unnormalized statistical models. In Proc. Conf. on Uncertainty in Artificial Intelligence (UAT), pp- 442 449. AUAI Press, 2010.
M.E Sander, V Roulet, T Liu, and M Blondel. Joint learning of energy-based models and their partition function. arXiv preprinl ar Xix::2200 18528, 2025.
D Siegmund. Importance sampling in the monte carlo study of sequential tests. The Annals of Statistics, pp. 673 684, 1976.
Y Song and D.P. Kingma. How to train your energy-based models. Xrviv preprint ar arXiv:2101.032 2021.
Y Song, C Durkan, I Murray, and S Ermon. Maximum likelihood training of score- based diffusion models. Advances Newral Information Processing Systems, 34:1415-1422 2021.
C Stein. A bound for the error in the normal approximation to the distribution of sum of dependent random variables. In Proceedings of the Sizth Berkeley Symposium on Mathematical Statistics and Probability, Volume 2: Probability Theory, volume 6, pp. 583 603. University of California Press, 1972.
W Stummer and I Vajda. On divergences of finite measures and their applicability in statistics and information theory. Statistics, 14(2):169-188) 2010.
15

---

Published in Transactions on Machine Learning Research (03/2026)
C Varin, N Reid, and D Firth. An overview of composite likelihood methods. Statistica Sinica, pp. 5-42, 2011.
P Vincent. A connection between score matching and denoising auloencoders. Newral Compulation, 23: 1661-16744 2011.
M.J Wainwright and M.I Jordan. Graphical models, exponential families, and variational inference. Founda- tions and Trends in Machine Learning, 1(1-2):1-305, 2008.
M Welling and Y.W Teh. Bayesian learning via stochastic gradient langevin dynamics. In International Conference on Machine Learning, 2011.
H White. Maximum likelihood estimation of misspecified models. Econometrica: Journal of the econometric socicty, pp. 1-25, 1982.
Xie, Lu, R Gao, and Y.N Wu. Cooperative learning of energy-based model and latent variable model via meme teaching. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Xie, Y Zhu, Li, and P Li. A tale of two flows: Cooperative learning of langevin flow and normalizing flow toward energy-based model. In International Conference on Learning Representations, 2021.
A Proofs
A.1 Lemma 2.1 Variational linearising the logarithm
Inspired by Jordan et al. (1999) and Ormerod & Wand (0010), show the following lemma:
Lemma 2.1. For all 2 0,
log = = min 2z + 11)
(8)
AeR
Proof. Let and R, define the function:
h(A) = ze A + x 1.
(31)
By differentiating this function with respect to X, we gel:
N'(A) - +1.
(32)
The differentiated function / is negative for X < log and positive for > logz. Thus the minimum of h is reached at log(2) and h(lg (z)) = log(z), hence the proof.
A.2 Theorem 2.1 Equivalence between SNL and the log- likeliidl
We begin by reminding notations: we consider an energy-based based model, which specifies a probability density over space X' through a parameterised energy function Eg X' + R. The associated density is:
Ex(x))
po (x)
(33)
Zo
where Zo = Bo is the partition function. Given data points Ii, In X, define the log-likelihood function:
((0) 1
Eo(x.) log Za-
(34)
We define as well the self-normalised log-liken liglliilloood (SNL) as:
{SNI (s,I,0,6)
Ee (xx) 6 --- + 1.
(35)
We now recall Theorem 2.1
16

---

Published in Transactions on Machine Learning Research (03/2026)
Theorem 2.1. For any given 0, when the SNL is marimised with respect to b, we have access to the eract log-likelihood of the model:
max (0, = E(0).
(11)
Moreover, at the oplimuum, b is the normalisation constant:
arg max BER
(S,I,(, b) = -lo Zn-
(12)
Finally, there is one-to-one correspondence between the local optima of the SNL and the log- boglikeelhhod.
Proof. Using Lemma 2.1 we show that for any 8, maXteR (0, E())
((0)
log Po (2:))
(36)
nI
log(E
Bo(ai)) log(e-Bo((u)) log Zo
(37)
1=1
log(E Fo(a))) log(e-Bo(ot)) min(6 (e
Zo + b 1)
(38)
π i=1
beR
1
log(e- (og(e-Bo(as)) (x(ss) + max(-aa-
maxx-e--22 b+ 1)
(39)
11
BER
max
log(e log(e-Bo(xx)) Bo(x.)) - - -bb Zp b b+1
(40)
max (8,
(41)
beR
We show that {()) and ESNI (0, b) share the same local maxima. Let 0* a local optimum of é, we will construct
a local optimum of ESNL. Let b* = log Z then
VolsNI(0" (0,)))(00 -1+ Zo =0.
(42)
Moreover,
VolsNI(O",0)
VoEo(zi)lo-- - Vo0000
(43)
n
VoEo -- Zo V log Zolo-
(44)
n
Eo 1 Zo V log Zole-
(45)
n
Zo.
VoLo Vo log Zolo-
(46)
Vo((0")
(47)
0.
(48)
Thus, for any local optimum of E(0*), the pair (0", log Zo-) is local optimum fSNL. Conversely, with the same reasoning, for any (0 local optimum of LSNL, 0 is a local maximum of f.
A.3 Theorem A.1 Concavity of the log- likelihood in exponential families
For completeness, we prove the classical result about the convexity of exponential families. For more details, see e.g., Wainwright. & Jordan 2008
Chapter 3).
17

---

Published in Transactions on Machine Learning Research (03/2026)
Theorem A.1. If (po)e is canonical exponential family, then {(f) is concave. In particular, the gradient and the Hessian of log Za are respectively the mean and the covoriance matriz of the sufficient stalistics.
Let us consider an exponential family whose densities with respect to a base measure are parameterised po(x) = ed s(x)=log 20, where s(x) is the sufficient statistics and 0 the natural parameters. To simplity formulas, will assume that we observe a single data point x E X. Observing several i.i.d data points will preserve concavity because sum of concave functions remains concave, so there is no loss of generality. The log-likelihood of such a model is given by:
Proof.
{ = 0" s(x) - log Zo = =0" - log / (0 dr.
(49)
We will prove that this objective concave by showing that the Hessian is negative semi-definite. Let's calculate the gradient and Hessian of log Zz. For integrals akin to the normalising constant, switching differentiation and integration is allowed (see, e.g., Lehmann & Romano 2022 Theorem 2.7.1), and we get
V log Ze Volog
(50)
[ (( c())
(51)
I e s(
(f do
(52)
[Bos(<))];
(53)
and
f ((0))(()) e gd((e) 1og) do (fols)e o(0)) 2os 20 (Vo log Zo)²
Ho Hlogog Zo)
(54)
{ ((D))(10)) od d(x) Zo dar
(55)
(fs())e (fstsseetttt))--eeeaaaaa 20 dx)
(f
1og) o(0) 2a do
T
(56)
=Eols())s(>>"} Eo(s())))o((())] -o(s(x))
(57)
(58)
Using the Hessian of log Zo. we can express directly the Hessian of the log-likelihooo E(0):
H(((0)) =ooos()))]
(59)
The covariance matrix Vo[s(z)] is positive semi-definite, thus the hessian H((()) negative semi-definite. Henco, E(0) is concave.
A.4 Theorem 3.1. Concavity of SNL in exponential families
Theorem 3.1. If (po)o is canonical exponential family, then (sN is jointly concave.
Proof. Using the same notations the previous proof, our exponential family is parameterised po(x)) e² s(x)-log Zo where s(x) is the sufficient statistics and the natural parameters. We again assume without loss of generality that we observe single data point x E X.
The selm-nordllised log-likelihood is as follows:
ESNL (0, b) = 8T sz=) - b-e -btlog Zo +1
(60)
8" s(x) - .b bols)
(61)
18

---

Published in Transactions on Machine Learning Research (03/2026)
Since the first term of the equation is affine, will show that the function (0. b) + e-btlog is jointly convex in (0,b) b). Let (b1, 01) and (b2, O2) any given pair of parameters and let À E [0, 1]:
f C a(x) (1-A))))
/ 2 be)))((())))(1-A)))) x(x) - dr
(62)
// (xe (xefF (1- A)e ((0)
((0)) -b dr
(63)
/ of (aaaaaaabiia + (1- 2) (1) (2 dd. dx.
(64)
The function (6.b) b) + bbbllog Zo is convex jointly in (6, (8,b), thus (6,b) + (SNL b) is also convex jointly in (0, b) which concludes the proof.
B The Gaussian case
We consider univariate Gaussian with unknown mean 0 R and known unit variance. The model is parameterised as an exponential family with energy and normalising constant:
1 02
Ea(x) Ozr log Zo
(65)
2
the base measure being the standard Gaussian measure. For a dataset (I), Zn) E R", the log-likelihood is:
((0)
1 n2 i=1
Σ
1 0.0 0²
(66)
2
which is concave and maximised at 0ML = En.
The SNL equals:
ESlL (0,b) =
Σ
0 b Zge -b +
(67)
i=1
n
r,0 6 b- -b000=b+
(68)
which is also concave and is maximised at (ÓSNI,, dSNL) = (Tin, (5in, 22/
C The Bernoulli case
In the same vein as in Appendix B we derive here the SNL for a Bernoulli distribution, in order to gain basic insights. We consider Bernoulli distribution with unknown natural parameter 0 e R (0 here is the logit of the probability of success). The model parameterised as an exponential family with energy and normalising constant:
Ex(x) = -0x, log Za = log (1 + 1+)).
(69)
the base measure being the uniform measure 10.1}.
For a dataset (II,,.,, Rn) E {0, 13", the log- ligeiiel is:
1 n Σ
3,0 log (1 + c1+))
((0)
(70)
which is concave and maximised at 0M.. = (x)) The SNL equals:
(0,L(
0.0 - b - Zge +1
(71)
t=1
x20 - b - - (1 1l+e" +e) +
(72)
which is also concave and is maximised at (0SNI,, dSKL.) (Tx) clgght(z))).
19

---

Published in Transactions on Machine Learning Research (03/2026)
D The Kullback-Leibler divergence for un-normalised densities
We consider measured space X, equipped with base measure da (typically the Lebesgue or the counting measure). Let SI, /2 be the densities of two finite measures. The Kullback-eibler divergence between these is then defined as
(T((111f2)= = log
-
[(((z)
Si (1))) f2(x)
(alsa)de [
(73)
It is clear that this reduces to the usual KL when fi and f2 probability densities. For more details, sec, for instance, Amari & Nagaoka (2000) Section 3.6) or Stunmer & Vajda (2010). Why is this sensible generalisation? We can write our un-normalised densities as fi = A1P1 and fz = P1P2: where
m= (1xx)x,, P2 = / f2(x)da.
(1
(74)
Plugging this into equation 73 gives
KL(fillf) (L (p2) + pn log
11
+ (12 p1)
(75)
H12
=1 |KL(PIlP2))
P2
(76)
P1
where bi:l> log Since M()) for all 1 and h(1) O, we will have
KL(fil122 0
(L(fill2)) = and only fi = fe.
This means that this generalised KL enjoys some of the nice properties of the usual KL, which motivates its use for statistical inference. Another interesting property that is direct consequence of Eq. 75 is that
KL(P1 (p2) min (L
(77)
which means that we can recover the KL between probability densities by minimising the KL between un-normalised densities, transforming the computation of the normalising constant into an optimisation problem. This justifies Eq. (19 Another interpretation of this property that the KL between pl and the set ccf2:>>>>> >> 0} is just the KL between P1 and p2-
The KL. divergence between two un-normalised densities relates to the self-normalised log-likelihood as such:
[L[paatal]ee lllleeee-pp -Bn) = / log (x) Eole)) Paata (data())(x)dzz + cZa -
(78)
--/ Ea(()) (x) logct + cZa - dr +
/ /log loo (Laa)) (x)dx.
(79)
does not depeud on 0 mor
As assume we have access i.i.d. datasel Fi, Tn; we estimate the above quantity:
Eo(a)) - logo cZo - 1 / log(Paattaz)) dx
KL(Pdatalles (Fo)= ~
(80)
[m1
-SNLL( log + log(pentu (2) (daata dx.
(81)
This implics that maximising the nolflnooma log- lieeliil will, asymptotically, resemble minimising the generalised Cullback-Leible divergence.
20

---

Published in Transactions on Machine Learning Research (03/2026)
E
Link with the Nonsker-Varadhar representation
E.1 The Varadh-- representation
The Vansker-aaradhan representation (Donsker & Varadhan 1975 provides a variational (Fenchel dual)
formulation of the Kullback-Leibler divergence between two distributions IP and B:
sup (RC) { 1 {1+
f h dPP -
f dBB..
()
(82)
where (RR) denotes the set of continuous bounded functions from Rd to R. The supremum is attained when h = log (ogg(d//a /dB), i.c., the log-density ratio between the two distributions.
This variational formulation has inspired several approaches to training energy-based models and defining divergences between distributions. Different methods arise from different choices of parametrisation or constraints the function h. In this section, we describe how SNL can be derived from this formulation and contrast it with two related approaches: Generalized Energy Based Models (Arbel et al. 2020 and KALE
Flow Glaser et al. 2021
E.2 From Donsker- Varaahan to SNL
To connect with the Donsker-
representation, we take P == Pdata (the data distribution) and define the base mcasure dB as the Lebesgue mcasure directly. Crucially, SNL does not optimise over all functions
h; instead we restrict the function h to be of the form h(x) = - Ee(r' (r) b where Eg is an energy function
parameterised by 0 and b E is a scalar. Substituting this into Eq..
82 yields:
f hade-
f dB =1
/ Eo(x) dP(x) - b c-b / E-Ball) dB (())
1+
(83)
/ Fo(x) -b b b-c -
(84)
where Zo = fe-bo()) Bo(xx) di is the partition function of the energy-based model defined by Eg. In our case, Eg is parameterized by neural network fo. When IP is the empirical data distribution, this becomes exactly the SNL objective (Eq. (10) in the main text):
a Σ
(0,
Eo(z,) - - -be e + +1].
(85)
Using a base distribution. The above derivation assumes that the function Eg defined with respect the Lebesgue measure, which not the case in the general setting when parameterized by an arbitrary neural network fo. To alleviate this issuc, introduced in Scction 2.4 fixed base density d(x)) and define the eased model as Eg(x) fxx - log d(xr).
From the Monsker-Varadlan perspective, this can be interpreted in two ways:
We can take the base mcasure B be the mcasure with density d(z) with respect to the Lebesgue measure dr, iee. dB(x) d(z)dar. In this case, the complete energy function Eg is still defined Ee(x) fo(x). The normalisation constant becomes Ze = - dB = ʃ ff(=)))((()) dr.
Alternatively, we can take the base measure to be the Lebesgue measure, and define the energy function as (x(x)) = fo(x) log d(xx). In this case, the SNT. objective obtained by substituting h(x)- -Eş(z) + log d(x:) into the Varadhnn representation, and the base density d(z) appears as multiplicative factor in the partition function Zg e fo(*) d(z) dB(x) = ----eeefeee --f(()) dr.
In both cases, we can use a proposal distribution g to obtain unbiased estimates of the partition function and gradients, without affecting the population-level objective or the correspondence with maximum likelihood. In particular, if one chooses proposal density (z)) be the base density d(x), we obtain a special case of SNT. loss that corresponds to special case of the GEBM.
21

---

Published in Transactions on Machine Learning Research (03/2026)
E.3 On the difference with Generalized Energy Based Models
The Generalized Energy Based Model (GEBM) is derived by using a slightly changed parameterisation. Instead, of having BB as the Lebesgue measure, the base measure B is learnable distribution, typically an implicit generative model such as a GAN generator. The GEBM objective is obtained by substituting h(x) = Exx) into the Donsker- Varkdhnn representation with this lcarnable basc B:
GEBM(E,,,B) = -L E(s) e f E(a)
dIB
(z).
(86)
In the general case, this base measure does not admit density with respect to the Lebesgue measure, and the energy function E is only defined with respect to this base mcasure.
No separation between base and proposal. A key distinction between SNL and GEBM lies in the relationship between the base mcasure and the proposal distribution. In SNL, the base measure d(z) is part of the model definition, while the proposal g(xr) used for importance sampling can be chosen independently the population-level objective invariant to this choice. In GEBM, no such separation exists: samples for estimating the partition function are drawn directly from the learned base itself. The base and proposal are thus identical, and both change during training. This coupling means that changing B simultancously changes both the probabilistic model and the distribution from which samples are drawn.
In the special case where the base distribution B admits density with respect to the Lebesgue measure, the GEBM objective can be understood as SNL where the base distribution and the proposal distribution are forced to coincide. However, in the general case where B is an implicit model (e.g., a GAN generator), this interpretation does not apply, and the lack of separation between base and proposal becomes a fundamental characteristic of the method.
E.4 On the difference with KALE Flow
KALE Flow (Glaser et al. 2021) also builds on the Varaddan representation, but takes a fundamentally different approach. Rather than evaluating the dual a fixed parametrisation, KALE optimises over a restricted function class H and introduces regularisation term:
KALE, (PI[B) = (1 + a)) sup
1 1
/ heT -harp-fe e JB 2
a 11
(87)
where is typically a reproducing kernel Hilbert space (RKHS) and a > 0 is a regularisation parameter. The KALE objective defines surrogate divergence between IP and B that depends on both the function class H and the regularisation strength a. For finite a and a restricted H. the minimiser of KALE does not, in general, coincide with the maximum likelihood solution. Instead, KALE is designed gradient flows: the optimal h" in Eq. (87) provides a transport direction that moves samples from B toward P. This makes KALE well-suited for particle-based based inference and sampling methods, where the goal is to iteratively transport a source distributiin loward a larget.
tool for defining
The key distinction with SNL is that KALE optimises over functions to define a divergence, whereas SNL fixes the paramctrisation = Eg and optimises over 0 and b perform maximum likelihood estimation. In SNL, the function h not a transport potential but rather encodes the energy-based model whose parameters we wish to learn. As result, SNL retains exact correspondence with maximum likelihood, while KALE defines relaxed divergence useful for gradient flows.
F Algorithms
Algorithm 1: Training an EBM for density estimation using SNL loss and proposal q.
input : Learning iterations, T; learning rate, vy initial parameters, bo observed examples, {m: )ar batch size, na; number of samples from the proposal g, M.
output: 0r, by
for = T do
Mini-batch: Sample observed examples {
2. Proposal sampling: Sample M elements from the proposal q((x))
Learn EBM parameters 0: Update 0++1 = 0. - VoVosN (0, b) using V VolsNI. (D..b) defined in Eq. 15
4. Learn d. Update bit1 b, v7VolaNI be) using VolsNu(B, be) in defined Eq. 116.
22

---

Published in Transactions on Machine Learning Research (03/2026)
Algorithm 2: Training VAE with EBM prior using the SNELBO loss.
input Leearning iterations, T; learning rale, v imitial parameters, (Oo, yo, coo, bu); observed examples, {ar)i=i: batch size, number of samples from the base d, M.
output: 0r, PT, br.
for = 0 1
1. Mini-batch: Sample observed examples far 1
2. Proposal sampling: Sample M elements from the base 7m ~ d(xm)
Learn EBM parameters 0: Update 0141 02 - 1 V C/Vooss..( (10, 0, using Eq. 288
4. Learn VAE parameters: Update {7, 0)41 {n. 6) n (v.o) (SNt)) (O, 7, o, b) using Eq. 28
5. Learn & Update bi41 b v,V,NNL (0, bi) using Eq. 28
G Derivation of the SNELBO
Using the variational distribution (z(22)), write the regular ELBO for the VAE with the energy-based prior as
log e-Bo(=)d(=) (-(2(2)) ( Ze
[(0, o. 7) Bg (=|x)) [o (og + (xlur)
(s/ur
(88)
which is a lower bound, 6) 2 _(0, 7), on the log- likelihood
E(0, Φ) (0)) po(r)l))oo=)) dz,
(89)
where we left out the sum over data to simply the notation. Using Lemma 2.1
we define the SNELBO as
log d(z) Øn (a (r)
CSNL. (0,0,7,6) Bg (=|=) (=1=) (x]=]] B (=|m)
(x Eg((2) - b] - Zoe + 1, (90)
which can be writlen using the base distribution d,
log d(z)
CSNI. (0,0, = B2g()l2) [[]122 E (elo))
4- (2|zz)
+ R (2
[-Fo(>) -b] -Ea(a) e Fa(=)-b) +1 (91)
Lemma 2.1 gives directly the following results:
((0, 2 ((0,0,)) (0,
(92)
H Regression datasets
13.
35
33
-25
-25
-18
(a) 1D regression dataset
(b) 1D regression dataset 2
Figure 5: Visualisation of the two toy regression datasets. The z-axis corresponds to the input of the regression (a in Eq. (27 and the y-axis corresponds to the regressed value (y in Eq. (27
23

---

Published in Transactions on Machine Learning Research (03/2026)
The first dataset set, on the left-hand side of Fig. is mixture of two Gaussians with weights 0.2 and 0.8 for negative values on the r-axis and log-normal distribution Log -V 0.25) for positive values of æ. There are 2000 training samples that we generated by uniformly sampling values in [-3,3].
The second dataset, on the right-hand side of Fig. 5 is defined for x in [0, 1] and is divided into four chunks. The first one, for < 0.21, is sampled from Be 0.5, 8 = 1); the second one, for 0.21 S x 0.47 is sampled from N (μ= cos 2.a [3 cos - 2)): the third one for 0.47
0.61 from an increasing uniform distribution; the fourth and last one, for 0.61 S S is obtained from a mixture of uniform distribution, U(8, 0.5), and u(54.5, 1.5).
Experimental setting
1.1 Multivariate von Mises
The multivariate von Mises distribution has density of the following form:
MuM(a)= Z8.A..
exp 1²" c(x, 0) + s(x, 0))
(93)
with Ze A an unknown normalisation constant, 8 the localisation parameter, A the correlation parameter and k the concentration parameters verifying: -π < 0; S T, < 11 S π, K; 2 0, -oo < >ij < 00 and A is such that (A)ig Xij- = Ajı; i i j. Aa 0. We defined c(r, 0) cos (æ: 01) cos (x: 02) ..... cos (x 0p) and s(z, 0)" (sin (or , sin (xr ........ sin (x: 07))
1.1.1 Implementation of the Multivariate von Mises To implement the multivariate von Mises (mVM) distribution, we adopt a parameterisation that ensures the required constraints the concentration and interaction parameters:
Concentration parameters are constrainod be non-negative. in the log-domain and applying the exponential function:
this by paramctorising
K₂ exp(n)) where 7): R.
Interaction matrix A must be symmetric to ensure valid dependencies among angular variables. We construct it by symmetrising lower-triangular matrix:
A A L=
where L is a lower-triangular matrix with unconstrained entries.
1.1.2 Sampling the dataset from MaM using Gibbs Sampling
This distribution has an unknown normalisation constant, which prevents the use of the inverse cumulative distribution function for sampling. In fact, there no direct way sampling from this distribution. However, the distribution of single dimension conditioned on the other lcads a simple von Mises distribution nM (pj-rest Ky-rest) with the following parameters:
(4) M.
M1 + tan
Σ 241
Xse sin (xe
0z)
/MSg
(94)
2
1/2
R
kig
+
Σ Ajesin (ars - Oi)
(95)
Sampling from a standard univariate Von Mises distribution Best & Fisher (9799 is possible. Thus, by successively updating the conditional parameters (Hi-rest Kij-rest ) and sampling the corresponding coordinate from the resulting von Mises distribution using Gibbs sampling, it is possible sample from the multivariate Mises distríbution.
24

---

Published in Transactions on Machine Learning Research (03/2026)
Following the experiments in Mardia et al. 2008 consider four parameter groups for the Multivariate Mises distribution that encompass different. scenarios. These parameters can be found in Table 17 For each group of parameters, we sample 100 data points by repeating 50 Gibbs sampling cycles over each dataset. We fix the location parameters to 0 and assume they are known.
1.1.3 Training Hyperparameters
For each set of parameters, we simulate five datasets of 100 samples with Gibbs sampling, where each sample is separated by 50 steps of filtering (following the setup in Mardia et al. (2008)). We report the average parameter estimate and standard deviation for parameters estimated with SNL in Fig, 2 and Table 17 We trained MvM with SNL for 10 000 steps using 100 samples from Uniform Proposal on the Torus using the Adam optimiser with learning rate of 0.1.
1.2
Density estimation with Truncated distribution
We are interested in obtaining the parameters of the constrained mixture of Gaussians in Fig. 3
1.2.1
Mixture parameterisation
We parametrise the multivariate Gaussian distributions to ensure the covariance matrix is symmetric and positive definite:
Diagonal elements of the covariance matrix are parameterised using log-scale transformation:
Eding exp(o):
where a E Rd is vector of unconstrained parameters.
Off-diagonal structure is captured via Cholesky decomposition LLT, where L s lower triangular.
The full covariance of single Gaussian of the mixture is obtained follows
Σ Eding + LLT, where L is lower triangular.
We train a mixture of K full-covariance Gaussian:
p Σ NxN(2 Ma, Ex),
(96)
which is then truncated with a known truncation h: + 10,1}:
PTruneated (I) x e Iaco)= Ekn (a
(97)
In practice, we use C 1e8 to enforce high energy outside the truncated domain.
We train the model using SNL with simple univariate Gaussian Distribution as proposal. We optimise the model using Adam Kingma & Ba (2015) for 10 000 steps and learning rate 0.01, and we chose C = 1e9 to mimic a potential barrier. The parameters are initialised using standard Gaussians and K-Means centres.
1.3 Density estimation
For density estimation, we parametrise the energy Eg using neural network with parameters given in Table
Ep
Activation
Output shape
Fully Connected
ReLU
200
Fully Connected
ReLU
200 100
Fully Connected
ReLU
100 50
Fully Connected
ReLU
50 x 50
Fully Connected
ReLU
50 x
Total trainable parameters
30450
Table 6: Eg for the toy distribution estimation
25

---

Published in Transactions on Machine Learning Research (03/2026)
1.4 Energy Based Regression
In Energy Based Regression, we consider an architecture similar to Gustallsson et al. (2020) which a feature extractor he is parameterised as neural network. The outputs of the feature extractor are fed both the energy Eg y) fo (he(2)) (z), with overhead fo parameterised by neural network. The normalisation constant ig obtained by considering neural network overhead over the feature extracior by bs((())).
When indicated, we use Mixture Density Network as proposal distribution. A Mixture Density Network (MDN) models a conditional distribution ( x) using a mixture of Gaussians, where the mixture parameters are predicted by a neural network conditioned on z. That is
K
( x) = Σ Te [x( (x)
(98)
where:
(2) are the mixing coefficients, satisfying <x(x) 2 0 and E TT(x) = 1,
(x)) IRd is the mean of the k-th component,
(x(z) e Rd is the standard deviation of the k-th component.
All parameters (Tk, Pk: Ex 1== are predicted by a neural network that use the extracted features h(z) input.
1.4.1
Toy regression
For Toy regression, we follow the same training procedure as Gustafsson et al. (2022) but replacing the NCE loss with the SNL loss. As such, the model is trained for 75 epochs, a batch size 32, 1024 samples and a lcarning rate 0.001.
For toy regression problems defined in Fig. with results in Table 20 and Table we parametrise the feature extractor he Table 7 the energy overhead Table S and the normalisation constant as Table 10 The mixture density network's parameters are given in Table 9
Feature extractor
Activation
Output shape
Fully Connected
ReLU
10 x
Fully Connccted
ReLU
10 x
Fully Connected
ReLU
10
Total trainable parameters
210
Table 7:
Feature extractor. Inputs and outputs h
Eg
Activation
Output shape
Input y + Output f(y)
Fully Connected
ReLU
16
Fully Connected
ReLU
16 32
Fully Connected
ReLU 32 x 64
Fully Connected
ReLU 64 x 128
Concalenation of hr and f())
Fully Connected
ReLU
144 10
Fully Connected
ReLU
10 x
Total trainable parameters
30450
Table 8: Eg for 1d regression estimation.
26

---

Published in Transactions on Machine Learning Research (03/2026)
MDN
Activation
Output shape
Input lig
Fully Connected
ReLU
10 x 10
Fully Connected
ReLU
x
Total Irainable parameters
100 10 K
Table 9: Neural network estimating one parameter of the MDN with components in the mixture.
b
Activation
Output shape
Input hg
Fully Connected
ReLU
10 x 10
Fully Connected
ReLU
Total trainable parameters
110
Table 10: Neural work estimating the normalization constant Za for every x.
1.4.2 Image for regression
For image regression, we follow the same training procedure as Gustafsson et al. (2022). Each model is run for 75 epochs, with batch size 32, learning rate 0.001 and M = 64 (unless specified otherwise) samples from the proposal distribution.
For image regression problems with results in Table and Table 21 the feature extractor Resnet-18 He et al. (0166 from torchvision Paszke et al. (2019). The energy overhead is parameterised as Table 11 and the
normalisation constant Table 13
The mixture density network's parameters given in Table 2
Eg
Activation
Output shape
Input y + Output fly)
Fully Connected
ReLU
x 16
Fully Connected
ReLU
x
Fully Connected
ReLU
32 x 64
Fully Connected
ReLU
64 x 128
Concalenation of h2 and (())
Fully Connected
ReLU
640 640
Fully Connected
ReLU 640 1
Total trainable parameters
420816
Table 11: Eg for 1d regression estimation.
MDN
Activation
Output shape
Input h
Fully Connected Fully Connected
ReLU
512 512
ReLU
512 K
Total trainable parameters
262144 + 512 x K
Table 12: Neural network estimating one parameter of the MDN with K components in the mixture. We use three such networks for Tus Paj Tg.
27

---

Published in Transactions on Machine Learning Research (03/2026)
EBM Model for BinaryMNIST
Layers
In-Out Size
Stride
Input: 2
100
Linear, LReLU
200
Linear, LReLU
200
Linear
Table 14: Architecture of the Energy-Basen Based Model (FBM) prior used for all the datasets in the VAE with
EBM prior.
b
Activation
Output shape
Input hg
Fully Connected
ReLU
512 512
Fully Connected
ReLU
512
Total trainable parameters
262656
Table 13: Neural network estimating the normalization constant Zz for every z.
1.4.3 Training the proposal distribution for MDN
Following the method from Gustafsson al. 2022 the MDN proposal is trained by minimising of the negative log-likelihood and the KL divergence with the EBM:
N
M
-Eo (o..s/)))) mm)
log
Σ
e
1
log uo (ur uoss Φ)
(99)
N
M
(m)
2
2m
m=1
1/i
This allows us to guide the proposal distribution towards the EBM. While this is trained at the same time as the EBM, only the head of the MDN is updated, while the feature extractor ((x)) remains fixed.
1.5
VAE with prior EBM
All the parameters of the VAE with prior EBM are given in Table 14
Table 15 and ??.
28

---

Published in Transactions on Machine Learning Research (03/2026)
Generator Model for BinaryMINIST, ngf 16
Layers
In-Out Size
Stride
Input:
16x1x1
4x4 convT(nnff x 8), LReLU
4x4x (ngf x 8)
xx3 convT(nnf 4), LReLU
7x7x (ngf 4)
4x4 conv'T(ngf 2), LReLU 4x4 convT((1, Sigmoid
14x14x (ngf 2)
28x 28x1
Generator Model for CIFAR-10. ngf 128
Input:
1x1x12 128
8x8 conyT(nnf 8), LRtLU
8x8x(ngī x 8)
4x4 4 (ngf x 4), LReLU
16x16x (ngf x 4)
2
4 4 convT((ng x 2), LReLU
32x32x (ngf x 2)
3x3 conv (33, Tanh
32x32x0
Generator Model for CelebA, ngf = 128
Input:
1x1x 100
4x4 convT(ngf 8), LRcLU
1x4x(ng) 8)
4 x 4 convT(nng 4), LReLU
8x8x (ngl 4)
4 x 4 convT(nng 2), LReLU
16x16x (ngf x 2)
2
4x4 cony 1), LReLU
32x32x (ngf x 1)
2
4x4 (3)) Tanh
64x64x3
2
Table 15: Generator architectures used for BinaryMNIST, CIFAR-10, and CelebA datasets. convT(n) denoles
a transposed convolution with n output feature maps.
Encoder Model for BinaryMMISS nef 16
Layers
In-Out Size
Stride
Input: æ
1x28x28
5x5 cngv x 2), LReLU
14x (ngf x 2)
5x5 conv(ng( x 4), LReLU
7x7 (ngl 4)
5x5 cngf x 8), LReLU
4x4x (ngf 8)
Linear
16
Encoder Model for CIFAR-10 (matches code), nef = 100
Layers
In-Out Size
Stride
Input:
ncx32x32
4x4 conv(nef), LReLU
16x16x(nef
4x4 conv(nefx2) LRcLU
8x8x(nef x2) 4x4x(nefx4) 2x2x(nefx8) 1x1x(nz)
4x4 conv(ne (uv(nexx4), LReLU
4x4 conv(ne x8), LReLU
4x4 conv(nz)
Flatten
n2
Linear
n1
Encoder Model for CelebA
(matches code). nef 100
Layers
In-Out Size
Stride
Input:
ncx64x 64
4x4 conv(nef), LReLU
32x32x (nef)
4x4
conv(nefx0) LReLU
16x16x (ne, x2)
2
4x4 conv(nef x4), LReLU
(nex(nefx41 (x4x(nefx8)
2
4x4 conv(nefx8)) LReLU
4x4 conv(ne 16), LReLU
4x4 conv(100)
1x1x(100)
Flatten
100
Lincar
100
Table 16: Encoder architectures used for BinaryMNIST, CIFAR-10, and CelebA. For CIFAR-10 and CelebA,
the table matches the provided implementation: stacked 4 x 4 Conv2d blocks with stride 2 and pad 1, followed by a final 4 x Conv2d to nz, flatten, and a linear layer.

---

Published in Transactions on Machine Learning Research (03/2026)
1.5.1 Sampling from the prior EBM
Samples from the EBM prior are obtained using Langevin dynamics with 80 warm-up steps, a step size of 10-1, and thinning set to 1, from which 64 samples are retained.
30

---

Published in Transactions on Machine Learning Research (03/2026)
Figure 6: Each row is dataset, the first column displays samples from the dataset, the second column displays the energy function of an EBM trained with the self normalised log-likelihood (SNL), the third column displays the energy function of an EBM trained with Noise Contrastive Estimation (NCE). We use a standard Gaussian the base distribution for both training methods. These parameterisations correspond to the first two lines of Table [
31

---

Published in Transactions on Machine Learning Research (03/2026)
E V
Figure 7: Each row corresponds to a dataset, the first column displays samples from the dataset, the second column displays the energy function of an EBM trained with the self normalised log-likelihood (SNL), the third column displays the energy function of an EBM trained with Noise Contrastive Estimation (NCE). We use the Mixture density Network (MDN) proposals with K 2 the proposal for both methods (see Table 3).
32

---

Published in Transactions on Machine Learning Research (03/2026)
J
Additional results
True
ML + SE
PL - SE
SNL t
SM -
K1
2.00
2.66 + 0.38
2.81 + 0.53
1.99 + 0.28
1.67 + 0.38
K2
3.00
2.84 + 0.39
2.81 + 0.44
3.06 = 0.23
2.92 + 0.45
K3
1.00
0.98 + 0.21
0.93 + 0.19
0.89 + 0.21
0.55 + 0.20
A12
2.00
2.33 + 0.55
2.64 0.80
2.31 = 0.48
2.78 - 0.28
A13
2.00
2.58 + 0.45
2.57 0.50
2.00 0.31
0.16 0.46
X23
2.00
1.49 + 0.48
1.17 + 0.51
1.82 + 0.32
0.89 0.55
K1
0.50
0.82 + 0.26
0.82 + 0.30
0.47 + 0.19
0.82 + 0.58
82
0.75
0.71 + 0.26
0.71 + 0.26
0.81 + 0.36
0.83 + 0.74
K3
0.25
0.39 + 0.26
0.40 + 0.28
0.13 = 0.13
1.53 0.86
A12
2.00
2.36 + 0.73
2.24 0.68
1.87 = 0.58
2.66 1.54
A13
3.00
3.27 + 0.71
3.36 + 0.64
2.91 + 0.63
2.95 + 1.41
X23
4.00
3.49 + 0.70
3.53 + 0.69
4.62 = 0.43
2.98 + 1.77
K1
2.00
2.63 + 0.97
2.65 + 0.98
2.19 = 0.92
4.44 + 4.87
82
2.00
1.66 + 0.81
1.65 0.85
2.04 0.27
5.99 7.31
K3
2.00
2.01 + 0.85
2.02 + 0.92
1.87 = 1.01
4.01 + 10.0
A12
20.00
36.85 = 8.63
36.76 = 6.99
15.6 = 8.51
8.6 = 12.76
\13
30.00
40.01 + 8.55
40.15 + 8.49
35.0 + 8.11
19.26 + 16.06
A23
40.00
23.66 + 8.61
23.64 + 7.87
40.23 + 6.88
26.47 + 19.97
K1
2.00
1.84 + 0.23
1.84 0.23
2.17 0.14
2.20 0.22
K2
2.00
1.83 + 0.23
1.83 + 0.23
2.10 + 0.19
2.14 + 0.36
K3
2.00
1.94 + 0.24
1.94 1 0.23
2.09 - 0.13
2.14 0.21
X12 0.10
0.15 + 0.28
0.14 + 0.28
-0.12 = 0.17
2.25 + 0.20
A13
0.10
0.17 + 0.28
0.16 + 0.28
0.01 = 0.19
2.29 + 0.29
A23
0.10
0.12 + 0.28
0.12 + 0.30
0.12 + 0.26
2.38 + 0.20
Table 17: Parameters estimates of multivariate von Miscs distribution (Mardia et al. sets of parameters. For this experiment, the location parameters are known and set to 0. The Maximum Likelihood (ML) and Seeudo-Likelhood (PL) results are directly reported from Mardia et al. (0008). these cases, the uncertainty estimates (SE) are obtained using Jacknife cstimators. The score matching estimators are obtained using explicit formulation in Mardia al. (2016) while the SNL estimators are calculated using gradient descents. We report the average parameters and associated standard deviation over five runs with five different datasets.
2008 for 4 different
C
T A 112
|| All
---- -
EM
4.934 + 0.419
0.0143 + 0.0084
10.87 + 11.22
55.66 + 35.61
GMMis
4.284 + 0.348
0.0043 0.0029
1.46 0.84
16.23 + 5.74
SNL
4.145 + 0.095
0.0066 + 0.003
0.87 0.83
17.89 +-2 4.38
Table 18: Comparative evaluation of the three estimation models (Standard EM Dempster et al. 1977 GMMis Melchior & Goulding (2018) and SNL). We evaluate the likelihood of the model on a non-tr non-truncated test datasct show the quality of the estimated parameters and the norm of the difference with the original parameters. Each entry reported mean standard deviation over five generated datasets and runs. Both SNT and GMMis perform par, outperforming the original FM, but it should be noted that GMMis is much faster than doing gradient descent with SNL.
33

---

Published in Transactions on Machine Learning Research (03/2026)
10a
EM
GMMIs
SNL
101
109
Values
10-1
10
10
-eest
Im
Rull
Ela
Aull 111z1 211112
Figure 8: Log-scale bar plot comparing EM, GMMis, and SNL methods for the truncated mixture of Gaussians across four evaluation metrics. Bars represent the absolute mcan values, with crror bars indicating standard deviation over five different runs and datasets. We report the negative log-likelihond over the untruncated test set and the norm of the difference of estimated parameters with the original parameters.
Dataset
VAE
SNELBO
short-term MCMC Pang et al. (2020)
CIFAR-10
107.57
98.36
70.15
CelebA
66.7
64.2
37.87
Table 19:
FID of generated samples for CIFAR-10 and CelebA. The results of short-term MCMC are directly reported from Pang et al. (2020). Though using the EBM prior with SNELBO improves slightly the generation, the quality ofthe generation is far from being competitive with latent short term MCMC.
34

---

Published in Transactions on Machine Learning Research (03/2026)
Models
Datasets
Regression Dataset
Regression Dataset
Objective
Proposal
b
Base Dist
f
ESNI.
fis
ESNI.
NCE
Mi, x
None
None
00.100
(L0-186)
0.638
108)
-2416
-3.049
9009
NCE
N(u Σ)
None
-0.336
(+0.468)
-15567
(+0.282)
-2.548
(+0.252)
-2.676
(m0.269)
NCE
(u, Σ)
MLP
None
-0.030
(#0.278)
-0.718
(#0.256)
-2.592
(+0.214)
-3.559 (-1.881)
NCE
(a, ))
MLP
0.614 (1o.6a2))
1.580
(10.480)
-2426
(10.257)
-2586 (..0236)
NCE
MDN K2
None
None
-05770 (#0 200)
-1.275
(#0.688)
-2.451
(#0.040)
-3.094
(=0.616)
NCE MDN K2
MLP
None
-0.611 (#0.164)
-1.492
(20.908)
-2.451
(20.088)
-2.634
(=0.084)
SNL ((, Σ)
None
None
0.091 (=0.122)
0.023
cto.0r1)
-1.597
(#0.047)
-1.619
(=0.063)
SNL
N(,,
None
0.065 (=9.084)
0.044
(20.095)
-1.493
-1.503
(=0.011)
SNL
(I,,
MLP
None
0.164 (20.058)
0.033
(10.077)
-1.813
(10) 1083
-1.36
(=0. (09)
SNL
(i, Σ)
MLP
0.091 (=9.094)
0.048
(+0.080)
-14668
(+0.014)
-1477
(+0.010)
SNL
MDN K2
None
None
0.227 (=9.058)
0,221
[#0.000
-2.061
(+0.145)
-2.70
(=0.143)
SNL
MDN K2
MLP
None
0.255 (10.0017)
0.251
(10.001))
2..99
(10.250)
2.170
(_0.353)
Table 20:
Evaluation of regression EBMs on the 1D toy regression problems with two different objectives and different sets of parameters. Each model is trained for five runs, and we report the mean and standard deviation of the estimated log- log-likelihood fıs and the self-normalised log-likelihood ESNL. Using the SNL as the objective clearly outperforms the NCE.
Models
Datasets
Steoring Angle
Goll Count
Uaces
HIWI
Objective
Proposal
Eas
Aus
fis
ASN
fis
Espa
NCE
Ni,.
-3.649
UNNORMALIZED
-3.967
-.,655
-3.147
-8
-11.02
UNNORMALLZED
NCE
MDN-4
-4.044 (46.740)
1002272 cea.nis
-3.556
(-2400)
UNNORSEALRED
33,876 c-a30))
-4,821
-12.093
(46.200)
UNNORSALLZED
NCE
MDN-8
-1.01
UNNORMALIZED
-3.561
UNNORMALIZZD
-1.123
-5.170
-11.96
UNNORMALIZED
SNL
N)
-2.653 (21.39
3,973 12110
-2.00 (2)
2.725
-2.66 ca
-2.91
-10.8
(21
-11.05 (23
SNL
Unikorm
-1.402
14423
-2.604
2 32620
-2,927
-2.965
-00.4
11051
SNL
MDN-4
-1,780 can-
1.795
-2.544 (22
-886
-2,022 o20
-3.004
10.08
10.11
SNL
MDN-8
1,633
1,602
2,501
2811
2.921
2.943
10.01
10.04
Table 21: Evaluation of EBMs for regression on image regression datascts with two different objectives and
different proposals. Each model is trained for five runs and we report the mean and standard deviation of the
estimated log-likelihood (lss) and estimated nelf-normalised log-likelihood (CSNz). When the proposal is MDN, the proposal is learned jointly with the model following Gustafsson et 2022
35


Reviews:
[{"review_id": "29fATORsDl", "reviewer": "Reviewer_sJRH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript presents a novel self-normalized log-likelihood strategy designed to improve likelihood estimation. The core contribution is the incorporation of additional learnable parameters to explicitly model the normalization term, leading to significant performance gains. Extensive experiments across diverse tasks convincingly demonstrate the method's efficacy.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors provide rigorous proofs for all theorems and statements in the appendix. The inclusion of comprehensive derivations and clear algorithmic descriptions significantly enhances the theoretical soundness and reproducibility of the work.", "audience_yn": "Yes", "audience_explanation": "The computation of normalization terms has long been a notorious challenge in energy-based models (EBMs). The strategy employed in this paper—approximating these terms via learnable parameters is highly valuable and offers a practical solution. Overall, I find the research meaningful and the findings significant to the field.", "requested_changes": "Although the authors acknowledge the method's limitations in high-dimensional scenarios, I strongly recommend applying it to the CIFAR-10 dataset and reporting the FID metric. Even if the results are not state-of-the-art, analyzing the performance on this dataset could provide valuable insights for future extensions of the method to higher-dimensional spaces.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "X7DyaEmBaK", "reviewer": "Reviewer_oAn8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This article introduces a self-normalizing likelihood estimation framework to train energy-based models when the normalization constant is intractable. The novel objective function provides a lower bound for the log-likelihood, and it can be optimized using stochastic gradient descent methods. The effectiveness of the approach is validated on various density estimation, parameter estimation and regression tasks. The effect of the dimensionality of an energy-based model is also discussed.", "claims_and_evidence_yn": "No", "claims_explanation": "The article is well written. However, It seems to me that the main idea of optimizing the SNL loss already exists in the literature, therefore some claims should be adjusted or clarified to have a better positioning. The evaluation of some numerical results could also be strengthened to make certain results more convincing.", "audience_yn": "Yes", "audience_explanation": "Although the SNL loss exists already, its combination with idea of importance sampling is interesting, and the results obtained from this study are worth publishing.", "requested_changes": "-\tThe loss l_SNL in eq. 10 seems to be the same as the KALE loss in Arbel 2021 et al. (eq. 10). Thus, one should clearly mention the results in this article, as l_SNL is the same as Theorem 2.1. The positioning in 2.5 related works should be made more specific about this point. \n-\tThe results in Fig 2. Evaluate the errors of parameter estimation partially, only with respect to the parameter kappa and lambda. It would be more complete if the error on the theta is also reported. \n-\tThe claim from Table 2 seems to be too strong. How do you conclude that the results outperform or are on par with other method? In particular, the MAE MoG provides a larger log-likelihood estimation -15.15 compared you EBM-SNL in Hepmass. This point should be clarified or reduced.\n-\tCould you clarify the Fig 5, what is the x-axis / y-axis? \n-\tAs Fig 2 in Gustafasson 2022, you could provide a direct 2d plot on the learnt energy to evaluate the results in 5.2.1 for 1d regression.\n-\tIs there any explanation why the method stops to improve after M=64 samples in Fig 4? Did you look at the lower bound l_SNL instead the upper bould I_IS? How l_SNL evolves with respect to M?", "broader_impact_concerns": "na", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hrd3ZPaYty", "reviewer": "Reviewer_DScY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new way to train energy-based models (EBMs) with (approximate) maximum likelihood without MCMC. The core idea is to introduce a self-normalised log-likelihood (SNL) objective by using a simple variational identity that linearises the problematic $\\log Z_\\theta$ term. Concretely, the method augments the EBM with one extra learnable scalar parameter $b$ intended to represent $\\log Z_\\theta$. The authors show that (i) for fixed $\\theta$, maximizing SNL over $b$ recovers the exact log-likelihood, and (ii) jointly maximizing over $(\\theta,b)$ recovers the maximum-likelihood estimate of $\\theta$ while also estimating the normalisation constant. :contentReference[oaicite:0]{index=0}\n\nA practical consequence is that SNL depends linearly on $Z_\\theta$, enabling unbiased stochastic gradient estimates via importance sampling from a proposal distribution $q$, and therefore direct optimisation with SGD/Adam rather than sampling from the model with long Markov chains. The paper also provides theory, including joint concavity for exponential families and an information-theoretic interpretation via a generalised KL divergence.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Overall, the main claims are supported by solid (and mostly clear) evidence, especially on the theoretical side and on the regression benchmarks.\n\n- Theory / objective claims: The paper’s central claims about SNL (i.e., (i) maximizing over the extra scalar parameter recovers the exact log-likelihood for fixed $\\theta$, (ii) joint optima correspond to ML optima, (iii) SNL is a lower bound and can be “sandwiched” with an IS upper bound) are stated precisely and backed by formal results and proofs (main text + appendices).\n- Optimization tractability claims: The argument that SNL enables unbiased stochastic gradients via importance sampling from a proposal $q$ is explicitly derived (showing unbiased estimators for $\\nabla_\\theta$ and $\\nabla_b$ under the proposal-sampling scheme), which is convincing as written.\n- Empirical claims: The experimental section provides multiple settings (directional/truncated, low-dimensional density estimation, regression, and VAE-with-EBM-prior). The strongest and clearest empirical support is for the claim that SNL training improves over NCE in regression/image regression, supported by repeated runs and tables/figures studying the role of the number of proposal samples $M$.\n- Limitations / scope: The paper is reasonably transparent that scalability to very high-dimensional settings remains an open issue.", "audience_yn": "Yes", "audience_explanation": "I expect a meaningful subset of TMLR’s audience would be interested", "requested_changes": "I do not have any particular requested changes. The submission is clear and technically sound as written; any further edits would be minor polish/clarifications rather than requirements for my recommendation.", "broader_impact_concerns": "I don’t see any specific broader-impact red flags in the submission: it’s primarily a methodological contribution (a training objective for EBMs that avoids MCMC by using a self-normalised log-likelihood and proposal/importance sampling)", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)