A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation
Authors: Jongha Jon Ryu
Abhin Shah
Gregory W. Wornell
Venue: ICML 2025 poster (2025)
Keywords: unnormalized models
exponential family distributions
noise-contrastive estimation
interactive screening
finite-sample analysis
Primary area: theory->learning_theory
TL;DR: We provide a new, unified view on learning unnormalized distributions via a family of NCE principles, and establish finite-sample rates for the wide class of estimators for exponential family distributions.

Abstract: This paper studies a family of estimators based on noise-contrastive estimation (NCE) for learning unnormalized distributions. The main contribution of this work is to provide a unified perspective on various methods for learning unnormalized distributions, which have been independently proposed and studied in separate research communities, through the lens of NCE. This unified view offers new insights into existing estimators. Specifically, for exponential families, we establish the finite-sample convergence rates of the proposed estimators under a set of regularity assumptions, most of which are new.

A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation
Jon Ryu'
Abhin Shah
Gregory W. Wornell
Abstract
NCE (f Cf-ConNNCE; Sec. 2.2). With this unifying view different estimators, clarify previously unrecog- nized and/or potentially misleading connections among existing estimators proposed for learning unnormalized distributions, as well as provide unified analysis.
This paper studies a family of estimators based on noise-contrastive estimation (NCE) for learning unnormalized distributions. The main contribu- tion of this work is to provide unified perspec- tive various methods for learning unnormal- ized distributions, which have been independently proposed and studied in separate research com- munities, through the lens of NCE. This unified view offers new insights into existing estimators. Specifically, for exponential families, estab- lish the finite-sample convergence rates of the proposed estimators under set of regularity as- sumptions, most of which are new.
2. Specifically, via the lens of a- a-CentNCE,, we reveal that several different estimators for learning unnormalized distributions can be connected and unified, including MLE (Fisher, 1922), MC-MLE (Geyer, 1994), and Glob- alGISO (Shah et al., 2023) as special instances. A local version of centered NCE estimators subsumes pseudo likelihood (Besag, 1975) and interaction screening objec- tives (ISO) (Vuffray et al., 2016; 2021; Ren et al., 2021; Shah et al., 2021a), which were proposed for learning exponential families corresponding to Markov random fields (MRFs).
1. Introduction
3. For f-CondNCE, we show that, contrast to the original claim in (Ceylan & Gutmann, 2018), the behavior of the f f-CondNCE estimator does not converge to the score matching (SM) estimator (Hyvärinen, 2005) in a small noise regime. In fact, show that the variance of f- CondNCE diverges in the vanishing noise regime, if the number of conditional samples is not sufficiently large.
Unnormalized distributions, also known as energy-based models, arise in various applications, such as generative modeling, density estimation, and reinforcement learning: we refer an interested reader a comprehensive overview paper (Song& & Kingma, 2021) and references therein. Such distributions capture complex dependencies and provide representational flexibility, making them attractive in fields ranging from statistical physics to machine learning. De- spite their widespread use, estimating parameters within these models poses significant challenges due to the in- tractability of their normalization constants.
4. As a concrete consequence of such connections, es- tablish the maite-sample convergence guarantees of the proposed estimators for learning bounded exponential family distributions, by building upon the analysis of GlobalGISO by (Shah et al., 2023). To the best of our knowledge, such guarantees are the first of the type for almost all the NCE estimators considered in this paper.
In this paper, we consider the problem of parameter estima- tion for unnormalized distributions, through the lens of the noise-contrastive estimation (NCE) framework (Gutmann & Hyvärinen, 2012). Our contributions are as follows:
1.1. Related Work
1. As variants of the f-NCE (Pihlaja et al., 2010) (Sec. 1.2), we study a family of NCE-based estimators, the a- centered NCE (a-CentNCI; Sec. 2.1) and f-conditional
While the celebrated maximum likelihood estimator (MLE), advocated by Fisher (1922), is arguably the de facto stan- dard for parameter estimation problems, is not directly applicable for high-dimensional unnormalized distributions due to the computational intractability of calculating the nor- malization constant. Several methods have been proposed as alternatives, including MLE with Monte-Carke approxi- mation of partition function (MC-MLE) (Geyer, 1994; Riou- Durand & Chopin, 2018; Jiang et al., 2023), score match-
'Department of EECS, MIT, Cambridge, Massachusetts, USA. Correspondence to: J. Jon Ryu <jongha.ryu@ corruu@@@mmi.ooo. gmail.com>. comb>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation
b (do; ya, - (o)) As (z)
(2)
Ben(e) [T( gdl
(a(zz) (x)
(x) }
(1)
(x)( (xn(())
= - 2
Runcerll" (po())po()))) B (ooc))) -
(2)
ing (Hyvärinen, 2005; 2007; Song et al., 2020; Liu et al., 2022; Pabbaraju et al., 2023), NCE (Gutmann & Hyvärinen,
straightforward.
The idea of NCE is to train the model do(x), so that it can be used to discriminate samples of the data distribution (a(a)) from samples of noise (or reference) distribution (n(r). A necessary condition for discrimination is that the support of gn. i.c., supp(gn), subsumes the support of (a(r), i.c., supp(ga). Hence, we define the (scaled) model density
2012; Pihlaja et al., 2010; Gutmann & Hirayama, 2011: I: Cey- lan & Gutmann, 2018; Uchara et al., 2018; Chehab et al., 2022; 2023), contrastive divergence (Hinton, 2002), among many other techniques. A comprehensive overview of these methods can be found in (Song& & Kingma, 2021).
For exponential families, there a specialized literature, with focus on learning undirected graphical models such as MRFs. In pioneering work (Besag, 1975), Besag pro- posed the so-called pseudo likelihood estimator, which can be understood as local counterpart of MLE. A recent and representative line of recent work includes ISO, GISO, and ISODUS, based on an estimation principle called interac- tion screening ( (Vuffray et al., 2016; 2021; Ren et al., 2021; Shah et al., 2021a). More broadly, for exponential family in general, Shah et al. (2021b), and in follow-up work with refinement in (Shah et al., 2023), studied variant of the interaction screening objective for training general expo- nential family without a local structure, which we refer to as GlobalGISO in this paper. We emphasize that these estima- tors have been proposed and analyzed in several different communities, and the literature lacks on a comprehensive understanding how different estimators can be compared. In this paper, our primary goal is to provide a unifying view on these different principles for learning unnormalized dis- tributions in a unified way via the NCE principle (Gutmann & Hyvärinen, 2012; Pihlaja et al., 2010).
db(xz) for a hyperparameter >>> 0, and we ratio (x(x)) vqq[=]]
wish to fit this the underlying density ratio ge(w)
(g) For
differentiable function h: 2 + R with Z c R we define and denote the Bregman divergence as
An An _ h(z)) h(zz) - (Vh(z'), 2 z')
for n, E Z, which is the approximation error of the first- order Taylor approximation of h(z) at z' For a given strictly convex function f: IR20 - R and a reference distribution gn (x)), we propose the f-NCE objective as in Eq. (1). The intermediate expression in Eq. (1) is used as conceptual device to derive the final objective in Eq. (2). We define the f-NCE estimator as a minimizer of the objective function:
07 (qd, 4n) E arg min CT (do: qd, 9n). 8EB
Given data samples ø1...., Ina drawn from (z) and noise samples x'....., xn,,, from go (2). the empirical estimator is 07 (Āa. 9n), where qu and ân denote the corresponding em- pirical distributions. We remark that directly inheriting the property of the Bregman divergence, the f-NCE objective is invariant to adding or subtracting a linear function and translation by constants; see Appendix 1.1.1 for a formal statement.
1.2. Preliminaries: -.Noise-Contrastiv Estimation
We consider an unnormalized density model {z)( 0 e 0 for a d-dimensional random vector & with support X' C Rd, where E IPP is parameter and θ C IP is the set of feasible parameters. Our goal is to find the best 0 E Ô that Φe (z) is closest possible the data generating distribution qa (z). We consider the well-specified case, where there exists 0* θ such that chg- (I:) x (a()..
By constructing the f-NCE objective in terms of Breg- man divergence, can casily prove that the objective is consistent in the population limit, which we call Fisher con- sistency, provided that the generating function f is strictly convex and the model is well-specified.
Proposition 1.1 (f-NCE: Fisher consistency). Let f: TR20 - R.. be a strictly convex function and assume supp(qa) supp(g.). If there exists 0 such that ce- (-) 9-(.), then (orp))
We start the investigation with an extension of the original NCE (Gutmann& & Hyvärinen, 2012), which we call f-NCE. This family of estimators was first derived (Pihlaja et al., 2010) in a rather convoluted way. Here, we introduce them as an instance of Bregman divergence minimization for density ratio estimation (DRE) (Sugiyama et al., 2012), in which way the consistency of the resulting estimator is
qa(-).
Remark 1.1. Since the original family of unnormalized distributions (Φe(x)) 0 E Q} may not contain normalized distributions, we consider an augmented family dg(x) & (<oo(x) for A (0,c) for c > 0 for J-NCE. Then, we
2

---

A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation
Table I. Examples of the NCE objective. Recall that 0 e (0, X
Name
Generator function (p)
NCE objective (()(
Log (Gutmann & Hyvärinen, 2012)
(p)(() - plogg (p + 1) log(p 1)
B [log Pe
Egg flog pg
Asymmetric power (a)
(p) =-=1
a(a-1] for $ 10,1)
T-a Rau 1( Iu )
1 + 1 gm
$2
A Asymmetric inverse log
fo(p) - lim (p) -logp
Reg mm + E. Dog 211
aj0
Asymmetric log
f1(p) - (fa + a = plog
Bgg log 2a + Egg 02)
att
mg
assume that {dg(z): O RJ is well-specified, i.e., there exists c* E R and 0r such that qa(-) - e" Φ0- (-). Hereafter, @ denotes the augmented parameter, where O without an underline denotes the original parameter.
y Pozza + (x)
44 Jo (x)]
Jo (x)1)
We consider the examples of f in Table as the canonical examples; each ʃ (or the corresponding /-NCE objective) named based on its correspondence to a proper scoring rule (Gneiting & Raftery, 2007). It is easy to check that v does not affect the objective function for the case of power scores fa (p), and we thus set
which we call the a-CemINCE objective. Here, note that the second term in the fa-NCE objective becomes con- stant, since we design the a-centered model such that Ey 1o (x)] = 1. Note that the expectation with respect to the reference distribution Jn is embedded in the normal- ization term of the new model. In Table 2, we provide a side-by-side comparison between fa-NCE and a-CentNCE objectives for (0, 13.
1 in this case. We note that in the DRE literature, similar objective based on the generator function (p) is known as Kullback-Leibler Importance Estimation Procedure (Sugiyama et al., 2008).
We define the a-CentNCE estimator minimizer of the objective function:
2. Two Variants of NCE
Ocen (qd- 4n) E arg min Lae (oho; ga, (n)-
In this section, we introduce two variants of the f-NCE framework: a-centered NCE and f-conditional NCE.
8EB
In this case, since any multiplicative scaling to Φo is canceled out in the centered model in Eq. (3), the Fisher consistency follows even when the model is well-specified up to constant, unlike the strict well-speciffednnss required in Proposition 1.1.
2.1. a-Centered NCE
Consider the asymmetric power generator function (p) for a E Th (with = 1); see the second row of Table 1. We will introduce a transformation called a-centering in Eq. (3), which normalizes a given parametric model do (x:) in an a- and gn dependenntt manner. Applying the normalized model to -NEN (i.e., NCE induced by the asymmetric power score) results in new variant of NCE. In Sec. 3.1, we show that this variant provides unified view several existing estimators, seemingly different at a first glance.
Proposition 2.1 (a-CentNCE: Fisher consistency). Let E R. Assume supp(ga) C. supp(gn). If there exists 0* and >
O such that cop- (-) = ga (a),), then (9))))))))))))))))))))))))))))) (-) x qa(-).
2.2. f-Conditional NCE
In the NCE literature, it is known that the noise distribution In must be carefully chosen to guarantee good convergence of the resulting estimator, generally considered hard in prac- tice (Chehab et al., 2022). Alternatively, Ceylan & Gutmann (2018) proposed a new framework called the conditional NCE (CondNCE), where the idea is to draw noisy samples conditioned on the data samples. CondNCE was further justified via connection to the score matching framework of Hyvärinen (2005). In this paper, we clarify the connec- tion to score matching (in Sec. 3.2), and establish the first finite-sample convergence rate of this estimator (in Sec, 4).
We define a normalized model of (x(x) called the a- centered model as
(x) A (o(x) (0)'
where
(0) A E (e) (n(z) jaj]/a
if + 0, if 0.
exp(EE (z) (log (a(e))
Note that Zo) lim Za (0). Applying the -NCE objective to the a-centered model, we define
Here, we introduce f-CondNCE, general CondNCE frame- work for convex function f. The idea same as f-NCE:
Ccent (0: qd: 9n) A Lace (poza; qd, 9n)
3

---

A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation
Table Special cases of the fa-NCE and o-CentNCE objectives. The view on the estimators highlighted blue and boldface via a- a-CentNCE are new; see Theorem 3.2.
Objectives
0
E + (log
$1 5n (Pihlaja et al.. 2010))
on 2(E (eNCE
Eg.lV )
Eng [log dn 1 Ban
de.
4g (Importance Sampling (IS) (Pihlaja et al., 2010; Riou-Durand & Chopin, 2018))
fa-NCE
(InvIS
(Liu et al., 2021))
Figs [+ Ie Dog 1
E,g [log do (MLE (Fisher, 1922).
+ log Em [
a-CeniNCC
Un 2B
E By
Iv
(GIobalGISO (Shah et al., 2023))
MC-MLE (Geyer, 1994; Jiang et al., 2023))
we aim to minimize the Bregman divergence between two density ratios with respect to f. In this case, instead of the noise distribution On. we consider channel (condi- tional distribution) [Vylz), and aim to contrast the joint distributions (a(z) vs. (a(y) (x)l)) Comparing to (z( vs. (2) in the standard NCE, the contrast is self-referential in the sense that the data distribution qa ap- (a(rr)(ylir) pears on the both sides. Let po(x, y) doty)=(=ly] be the model density ratio in this case, implicitly assuming -. We define the generalized conditional NCE objec- tive cgnd( qa, T) as in Eq. (4), where the last equality follows from po (y) po(x. - For further simplicity, we focus symmetric channels, i.c., TT)]2z) x (xly)), in
3.1. MLE, MC-MLE, and GlobalGISO as Limiting Instances of Centered NCE
As alluded to above, a-CentNCE estimators interpolate be- tween MLE (Fisher, 1922) (a = 1) and GlobalGISO (Shah et al., 2023) (a 0, specifically for exponential fam- ily), provided that Za (0) can be computed analytically, i... without estimation. In the case of estimating Zaa (f) with samples, a-CentNCE objective recovers MC-MLE (Geyer, 1994) when a = 1. We formally summarize the connections in the next statement and Table 2.
Theorem 3.1 (a-CentNCE subsumes MLE and Global- GISO). The following holds:
which case the ratio simplifies to (0(x),)) = (e(=) For
1. (a = 0: GlobalGISO) For an exponential family de (z), if X is bounded and (x) is a uniform distribution over X, the O-CentNCE objective Co qa, 9n) is equivalent to GlobalGISO (Shah et al., 202
supp(ga) X Rd, canonical examples are (f) a Gauss- ian noise < / Nyy: z, 121) and (ii) uniform noise over a f,-norm ball or sphere for some 2 1. We define the J-CondNCE estimator as minimizer of the objective:
2. (a = L: MLE) If Zı (0) is assumed to be computable for each 6, the I-CentNCE objective (6; qa, qn) is equiva- lent to MLE (Fisher, 1922).
Oppnd (qa, T) E arg min 27nnd (ó9; qd. T).
Similar to a- C-CentNCE, the Fisher consistency follows even when the model is well-specified up to a constant as any multiplicative scaling to de(x)) is cancelled out.
3. (a 1: MC-MLE) If (i) = Bg, 9.(2) is estimated with empirical noise distribution án (I)), the I-CenNCCE objective É (0;0) qd, qn) is equivalent MC-MLE (Geyer, 1994).
(a(ar)
Proposition 2.2 (f-CondNCE: Fisher consistency). Let be strictly convex function. Let =(y(z) be a conditional distribution such that (upp(e((z)(()lz) supp(es (=))))) If there exists unique 8* and c 0 such that cóp- (-) qa(-), then @oppe (gnn)) x (.(.).
Remark 3.1. Note that the connection between GlobalGISO and MLE can be made for the case when Zo( is assumed to be computable for any 0. At one extreme when 1, in which case the objective boils down that of MLE, it is clear that Z1( = E,,, ( = f (o(x))dz becomes the standard partition function. In the other extreme case where a + if do(xx) exp((0, (xx)) an exponential fam- ily, computing Zo( boils down to computing gyn. [xx()) since Zo(0) x cxp((9, (0,, Egg. (V1)). For special choice of qo (e.g., uniform distribution) and 1i (e.g., polynomial and si- nusoidal functions), this term can be computed analytically, as concretely illustrated by (Shah et al., 2023). We also provide an alternative theoretical view of the 0- O-CenINCE objective as certain KI. divergence minimization problem,
In practice, given na samples {<::) pM drawn i.i.d. from
(1()) and conditional samples (vig } conditionally in- dependent from क (y)[x;) for each i, we let ggond (ou; ea, #) denote the corresponding empirical objective with slight abuse of notation.
3. Connecting the Dots
In this section, we explain how the estimators introduced in the previous section unify and generalize the existing estimators and provide new theoretical insights.
4

---

A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation
(a(x) (o())<(((a)
)1 B (c) ( (a(y)<(zly))
(g (cho: qd, πT)
E (w))
A
((()))()l))) (o())=(<))))
IE (())m)(( (- (((((()))) ( [poly,
(4)
generalizing the justification for GlobalGISO given in (Shah et al., 2023); see Theorem B.1.
where y _ x + ev. Then, we show that the f-CondNCE ob- jective behaves as the score matching objective (Hyvärinen, 2005) in the limit of + O. Formally:
Next, we provide a result connecting fa-NCE and a- CentNCE estimators, under the assumption that we have an optimization oracle that finds the global minima of given objective.
Theorem 3.3 (Asymptotic behavior of population [ S-CondNCE for small c). The population f-CondNCE ob- jective can be written as
Theorem 3.2 (fa-NCE and a-CentNCE estimators are equivalent). For a set A C R the augmented pa- rameter space, let Ale & (0: (0, v) E A for some v E RR) denote the subset corresponding to O. Then,
Ci qd gsi = f1)
o(E²),
where
LST (Sm((bi A (gy(a) tr( tr(V's log (1)) VV log do (x))))
arg min 2 (xx))
Cree(0: qd, (n) arg min Ceent (0; qd: Qn)- d
denotes the (population) score matching (SM) objec- tive (Hyvärinen, 2005).
Remark 3.2. We remark that, for a =1 1, Riou-Durand & Chopin (2018) proposed to convert the MC-MLE objective by the inverse of the l-centering operation, which they call the Poisson transform (Barthelmé & Chopin, 2015), into the fi-NCE objective, which they call the importance sam- pling (IS) objective. In this view, a-centering be understood as the inverse of the generalized Poisson trans- form. Via the equivalence, Riou-Durand & Chopin (2018) analyzed the asymptotic property of MC-MLE by studying the JN-NC. Similarly, one can analyze the statistical prop- erty of GlobalGISO (with any valid choice of gn beyond the uniform distribution) when Zo( is estimated with samples from qn (x)) via analyzing the fo-NCE objective.
This statement generalizes the result in (Ceylan & Gutmann, 2018) for flog- -onddCCE to f-CondNCE for any f. Below, we explain why this statement may be misleading as the f T-CondNCE estimator with + 0 does not behave like the SM estimator. To correctly understand the behavior, we need to consider the empirical f f-CondNCE objective function that defines the empirical estimator, instead of the population objective.
Theorem 3.4 (Asymptotic behavior of empirical f f-CondNCE for small e). The empirical f f-CondNCE objective can be written as
Egnnd (de: qd. ab) = =-(11) f(1)
3.2. Revisiting the Connection Between CondNCE and Score Matching
+ (L) V log de Toje = da (6)²² 2 + o(e²).
Ceylan & Gutmann (2018) argued that for continuous domain X. the original CondNCE objective is related to the score matching objective of Hyvärinen (2005), justifying the consistency of CondNCE. Here, we demonstrate that this interpretation can be misleading in a realistic setting with finite samples. To revisit this connection. we further restrict the type of channels to ne (y)2z) parameterized by parameter 0, such that y Te (3)22) is equivalent to + ev for some ~ (-() with zero mean and identity
Here, we define the empirical sliced SM (SSM) objec- live (Song et al., 2020)
2 (0o: G.) ②b)
1 V log 60 (x:))
Ba(t)aa(o) U V log (o()) ( +
2
Remark 3.3. Since we assume that (6(v) has zero mean, Theorem 3.3 readily follows as a corollary of Theorem 3.4. as the O(e) term will converge to O in the population limit of 9s- In finite-sample regime, however, the dominating term of the f f-CondNCE objective becomes the O(e) term, i.e., as e + 0, we have
covariance, i.c., Eg, [v] = 0 and [oe]] la. With this simplification, we denote the objective function as
C
(cond (do;; qds +
(e) (- (x,
1 frand (do: qd. ②b) + f11 - Ex((2) V log (0(x))] Eg()))
+ po(y. (po(y,))) f())),,))))
(1)
5

---

A Unified View on Learning Unnormalized Distributions via Estimation
K=1
K=4
K= 16
k= 64
x10
Thus, the f-CondNCE objective is dominated by this statisti- cal noise term when с < 1 with fixed sample size of u ~ 9s and thus too small e should be avoided in stark contrast to the proposed justification in (Ceylan & Gutmann, 2018). We revisit this degrading behavior after the finite-sample guarantee of f-CondNCE in Remark 4.3.
10 10
x10-
1=10-0
It is worth noting, however, that Egg. [. gets more concen- trated around 0 as the number of slicing vectors increases. Therefore. one could consider a carefully chosen с as a function of the number of slicing vectors and distribution- dependent quantities, so that [Eng())d VV log (o(x)) Tu] still vanishes as the number of slicing vectors increases. In this way, the f-CondNCE estimator might be still consistent with small emulating the behavior of SSM
x100
M
6=10--
6=
A||
x10
E-10
Simulation. To demonstrate this behavior, we considered a simple synthetic setup, where the data generating distri- bution is N(u, 1) with μ = 1. With a conditional noise distribution π ( [z] = -N(y),,,11 with varying e, we plot the derivatives of the empirical objective of the original CNCE with varying K E {1,4, 16, 64], where the sample size is N = 10². As shown in Figure 1, the empirical deriva- tives characterize the mean fairly closely when e 2 100- or when e is small and K is large. This simple ID Gauss- ian example clearly shows the undesirable behavior of the CNCE objective when e is small. More ie-depth study on the effect of e and K for high-dimensional problems is left as a future work.
x10 4
1-10-2
1=100
5
5
5
Figure I. Derivatives the empirical CondNCE objective with varying E {10-10 ..... 10 and K E {1,4, 4, 16, 64) for ID Gaussian data with true mean μ = 1.0 (vertical dashed red lines) and conditional noise distribution <-y[=) A (vll,,
4. Finite-Sample Analysis
In this section, we provide finite-sample guarantees of reg- ularized versions of the aforementioned NCE estimators, specifically assuming an exponential family distribution model áo (zr) = exp( (0, (xr))) Here, 0 E R2 denotes the natural parameter, v:: X + R2 denotes the natural statistics, and p denotes the number of parameters. In what follows, we assume both well- well-specifiedness and identifiability, i.e., there exists unique 6* θ such that do- (-) ga(+)-
X' may naturally be truncated during data acquisition (Liu et al., 2022). Assumption 4.1 (Bounded maximum norm of vb).
supsex [x)]= Vmmax for some tmax O.
Assumption 4.2 (Bounded parameter space). For some
constant r > 0, supaee R(0)= Sr.
Below, we establish the parametric error rate 0/221/2) of convergence for the regularized NCE estimators. The proofs adapt the analysis in (Shah et al., 2023) for GlobalGISO, which in turn built upon (Negahban et al., 2012; Vuffray et al., 2016; 2021; Shah et al., 2021b). We note in passing that the non-regularized NCE estimators can also be ana- lyzed, but can only prove suboptimal rate of 1//4/ by following the existing analysis in (Shah et al., 2021b).
We note that the gradient and Hessian of the f-NCE objec- tive can be written as
(o V Epper (0) E.g. (a (po)] + Egg, (ii,,,,, (po)),
1
1
V2 C2C (0) = Egg [RiNO ( (.2e,, (po)) )] + E [ S.ce (po)],
where the functions c(i) (. (p) for i E {1,2} and E (d, are defined in the leftmost column of Table 3; see Lemma B.3.
Following (Shah et al., 2023), we are specifically interested in the case where the statistics are bounded and so is the parameter space. We note that the bounded statistics may not be too restrictive, as in many practical scenarios the domain
Our analysis relies on the boundedness of the model density ratio po E (Pmins Pmax). In each result, we clarify the defi-
6

---

A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation
nition of the worst-case density ratios (Pmins Pmax). These ratios affect the convergence rate through the following quantities:
We state the finite-sample statement with a minimal of as- sumptions, along with the bounded statistics and parameter space assumptions. While achieving the parametric rate of convergence ((2) is appealing, to have non-vacuous rates, however, we need all the quantities in the sample com- plexity expression to be within a range bounded away from or co. More concretely, if we further assume that the dual norm of the statistic supxex R (v)()) S is bounded for some constant
1 fr inf PE(pmin. Praae))
Snce,,, (p) and
(5)
(5) far PElPniia-Paans)
(1) (nce.f.r(p) for E (1,2),
sup
where E n). We remark that these quantities differ for each estimator. For the canonical choices of f(p), i.e., log and asymmetric power, these quantities are explicitly given in Table 3.
0, it is easy check that the worst-case density ratios are bounded as (Pmin Pmax) C (e-TT,e"7) for J-NCE, where is defined to be the diameter of ( mea- sured in the norm R(-); see Assumption 4.2. We note that the worst-case density ratios affect the quantities in Eq. (5) polynomially for the canonical examples in Table 3, which in turn affect the sample complexity polynomially, Hence, the leading constant grows exponentially in r and sim- ilar to (Shah al., 2021b: 2023). This remark remains valid for the following two statements for a-CentNCE and (7) f f-CondNCE, as the worst-case density ratio bounds depend similarly on T and т. We also remark that the minimum eigenvalue conditions are rypically assumed in the exist- ing finte-sample analysis (Vuffray et al., 2016; Shah et al., 2021b; 2023), while (Shah el al., 2021a) establishes an explicit lower bound on the minimum eigenvalue for node- wise-sparse Gaussian MRFs.
Let R: θ + R20 be a norm over θ, and R θ + R be its dual norm. Define
sup ||0||
7122
(6)
||0|12"
0E40) (0)
(0)
YR
AER ((0) | max
sup
R(0)
YR.2
A sup
(E)
||0||2"
(8)
Here 40 & {40: 0 E }}. These quantities capture the geometry of the norm R-) imposed on the parameter space 9, and appear in the convergence rates.
Theorem 4.2 (a-CentNCE: finite-sample guarantee), Pick a E R. Define
Theorem 4.1 (f-NCE: saitle guarantee). Pick a strictly convex functiin f: R2+ + IR. Define
(Piins Pinax)
inf (o:a(z), sup eo:a (x)
(DminsPmax)
( inf =EX.0EBXX
po(x), sup
po(r))
=EX aEe
=EX,
and define the quantities in Eq. for f = fa accordingly.
and define the quantities in Eq. (5) accordingly. For E {d, n), define
Let Pā-;o (x)
and let
Enn
yaj
A Ance
Amnin (Eg, (1bbbT))
X cent min,d
A Aumin (Beg [(v - Egg 18788 (a - B.g (o 1)
R. Let Peo..
be such that
xeent
Aamin (Eeg, (Rhe) pā-:al E.g (o. 18:00 IT).
Let Grent, R be such that
f.nsst E arg min de@
cyca (0; Gd. Qn) + Ann R(0)
Gcent.R arg argmin OE0
Caent qd qn) + Ana R(0)
for some Ana.n > Then, for any A and 8 E (0, 1), there exists choice of Amn.n, such that [ J.nann
f¹ ||2 I
with probability 2 ö, provided that for each r E nd.
for some And aalmmaa mml. Then, for any O and E (0, 1).
Define i'aax P t'max +
there exists a choice of Xny such that @ 0* ||2 < A
f,ma
(1 22 R. -=-==aa :00 22
f.r.
mmax
with probability 1 - 8. provided that
n4 = Ω max
A 12 nice,.dd mimdd + f.ce fnn fonn
(2)
mimnn )2
(mmax)
(B (B
Bnne.fa 3 IR- :0c max.a
2/2
2/ 2bil max p²
ni = 2 max
log
A (b fu )2((1 - acent + a cent 12
(Ance min.r
)
minn
1. vist max.a a (Acent )
log p²
Remark 4.1. To the best of our knowledge, this result is the first finite-sample convergence rate for F-NCE estimators.
7

---

A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation
Table 3. Definitions of e (o)), for i E {1, and E fd, n} for example generator functions f.
Definitions
Log
Asymmetric power
f(p)
fro
fa(p)
S.ce.f.a (0) -pf (p)
e(1)
-p"-1
enco f.n. (p) _ (D)
pa
p+1
e(2) (p) $pgs(p)
(1 - a)pa-
S.ce.f.a
(o-1)²
S.ne. (p) - *f( grl)))
(p+1y²
ap"
(B I B (nn
(1, 1) 1)
(pmin P
(B (B (32) Bac))
(1 a alpmin /ct pmnxx)
(b (co
(re, x). where x
Pmin
Pman (Pmax +1)2
a allpmxx: 100 Pmnin
[Pminn
Remark 4.2 (Special cases). For a O, this result gener- alizes the finite-sample analysis of GlobalGISO of (Shah et al., 2023) beyond when On is the uniform distribution. For = 1, we establish the convergence rate of the MIE, which we believe to be the first result of this kind.
é Amiia (Em(t)) (a
a 0. In The- orem C.3 in Appendix, we establish that the asymptotic covariance of the estimator is Viann a I, C where
(()))((()) - +(1)) I - Emg, (2.a)) J" f -
For the CondNCE estimator, we consider K = 1, i.e. we have {z,, y:) H qa (z)r() for simplicity.
Cj a Egg (x,z) Iscond (pg. (bb()) v()))(((() - +(v))"),
Theorem 4.3 (f-CondNCE: finite-sample guarantee). Pick strictly convex function f: R+ + R. Define
(x, y) - (a(x)= For a channel y ~ (lyaz) defined as y = x + Eu as in Sec. 3.2, it is easy to check that lime- IT,,
where we let q
Pmin
inf (ea))
po(a, u),
lim.-s0 2.G. Hence, in the small-c regime, the asymptotic covariance of the J-CondNCE also behaves as Vxcond 2 and hence blows up as e + O. These observations are consistent to Theorem 3.4.
Piax
sup (z.p)
po(x,
and define the quantities in Eq. (5) accordingly Let
Xcond
A Amiu
Proof Sketch. Our finite-sample analysis of the regularized NCE estimators follows closely that of Shah et al. (2023), which relies the seminal result of (Negahban et al., 2012) for regularized M-estimators:
Let goond.R be such that
fime
goond. Time
arg min BE
{ [ (6: qa. + Am R(0)}
Theorem 4.4 (Negahban et al., 2012, Corollary 1). Let 21...., 2N be i.i.d. samples drawn from a distribution p(z). Let hz(2) be a convex and differentiable function parame- terized by 0 E O. Let (0) & Ei= ho(z)) denote the empirical objective function. Define
for some Ans- > O. Then, for any A > 0 and 8 E (0, 1),
there exists choice of Am such that [
0+ 112 < A
with probability 21 1 5, provided that
(Boond.ra s.d + fond 22 R 200 22
(!))
ni 2 max
On arg aar min (or))n(_.))))) + R(O) (x,(0)+
(9)
(b + (/2) Dcond.f.nl f.nndd (A cond
Ao max (Acond)
log p² 8
where An is regularization penally and R: Q - R2zo is norm over θ Let 0* E arg ming sha=))/((2)) Assume that
Here, beond.far and B(i) f.. are defined similar to Eq.. (5), where the infimum and supremum are taken over |umin (max) ) in place of (Pmin: P.ax).
The regularization penalty An satisfies An 2 2R (Vo, where R*: - - Rzo a dual norm of R over the dual space O+;
Pmaxe Pmntn
Remark 4.3 (Behavior of f-CondNCE in small-e regime). As alluded to in Sec. 3.2, the undesirable behavior of f-
2. The empirical objective + (.(()) satisfies a re- stricted strong convexity condition at f = 8 with curvature > 0, i.e., A (0, (+) 2 k| 0 1
CondNCE with small e can be also seen from the sam- ple complexity, since the minimum eigenvalue Xcond R
8

---

A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation
Then, the estimator On in Eq. (9) satisfies
dimension. In such scenarios, it is convenient to learn the conditional distribution for each node rather than learning the joint distribution over all nodes. There exists long line of work on this approach, e.g., see (Besag, 1975; Vuffray et al., 2016; 2021; Shah et al., 2021a; Ren et al., 2021), a representative of which is the pseudo likelihood estimator of Besag (1975). Maybe not very surprisingly at this point, if we apply the NCE framework in local manner, it provides unifying view all of the aforementioned works. We defer a detailed discussion to Appendix E.
00,, - 112 S 3 An
7R.2-
K
To ensure the first condition with X, sufficiently small, we show that, with high probability, the gradient of the em- pirical objective is sufficiently small, using Hoeffding's 's inequality under Assumption 4.1. For the second condition, we show that the lowest eigenvalue of the Hessian of the empirical objective is lower bounded, again by Hoeffding's inequality invoking Assumption 4.1 and the positivity of the minimum eigenvalues of some second moment matri- ces. Combining the two gigbbbrbbability events by a union bound completes the proof.
Optimization Complexity. So far, we have focused on the statistical properties of the proposed estimators. Now, we make a few comments regarding the optimization com- plexity as concluding remarks. The first-order important property regarding optimization is the convexity of the ob- jective functions with respect to the natural parameter 8. In Appendix F.1, we characterize a sufficient condition for the convexity of f-NCE, a- a-CentNCE, as well as f-CondNCE. Specifically, we show that flor, and fa for a E [o, 1J result in convex objectives. Somewhat surprisingly, a counterex- ample of convex f which cannot guarantee convexity of the objective function is fa( for a & [0, 1].
Simulation. We include preliminary simulation result of some NCE estimators in Appendix G. We leave a more thorough empirical investigation on the estimators in this paper for high-dimensional problems as future work.
5. Discussion and Conclusion
Beyond Bounded Exponential Families. An intriguing question is whether we can relax the boundedness assump- tion on (xx), making our estimators applicable beyond bounded (or truncated) exponential families. Here, we high- light what we need to modify in the proofs to extend the validity beyond this assumption, using S-NCE estimators as an example. As sketched above, the proof of Theorem 4.1 consists of two parts: (1) the concentration of the gradient of the empirical objective around 0, at the true parameter Hr (Proposition D.I) and (2) the restricted strong convexity canticooncentration of the Hessian) of the empirical objec- tive, around the true parameter 8* (Proposition D.2). In- voking the uniform bound via the worst-case density ratios, we apply Hoeffding's 's inequality using the boundedness of the max-norm of (zr). For unbounded sufficient statistics, we need technique to handle the concentration behaviors, without worst-case density ratios bounded away from 0 and 00. For example, if the exponential family distribution is uub-Gaussian and the sufficient statistics are polynomials, one could use the nei-Weibull concentration bounds.
In the optimization community, recent line of work (Liu et al., 2021; Lee et al., 2023) studied the optimization land- scape of the original NCE objective and showed that the landscape can be arbitrarily flat even for a scalar Gauss- ian mean estimation. This is mainly due to the unbounded and light-tailed nature of Gaussian distributions. Under the boundedness assumption, we prove in Appendix F.2 that the empirical f-NCE objective function, for example, is smooth with probability 1. Then, from (Agarwal et al., 2010, Theo- rem 1), and the restricted strong convexity (Proposition D.2), a projected gradient descent algorithm has a globally ge- ometric rate of convergence. A recent work (Jiang et al., 2023) analyzed the optimization landscape of MC-MLE and proposed an optimization algorithm with efficient optimiza- tion complexity guarantee together with a strong empirical result, missing the connection to the original work (Geyer, 1994) and its statistical properties analyzed in (Riou-Durand & Chopin, 2018). Building on top of our work and (Jiang et al., 2023) could be an exciting future direction at the intersection of statistical and optimization complexity for learning unnormalized distributions.
Local Versions of NCE-based Estimators. So far, we take global approach to learning the parameter 9 by treating it as single object. In the context of exponential families, this is beneficial when exploiting a global structure on 0 such as a bounded maximum norm, a bounded Frobenius norm, or a bounded nuclear norm when 9 is matrix-shaped (Shah et al., 2023). However, for exponential families corresponding to node-wise sparse Markov random fields (MRFs), the structure to be exploited is inherently local. Specifically, in node-wise-spare MRFs, the conditional distribution of each node given all the other nodes can be expressed by number of parameters which scale with the maximum-degre of the MRF, which is assumed to be much smaller than the
Conclusion. We hope that this work offers unifying per- spective on both existing estimators and those yet to be discovered, and that it contributes to more systematic understanding of the trade-off between statistical and op- timization complexity in the context of efficient learning with unnormalized distributions. As emphasized through- out the paper, further investigation is warranted to better understand the empirical behavior of different estimators in high-dimensional settings.

---

A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation
Acknowledgements
Geyer, J. On the convergence of Monte Carlo maximum
likelihood calculations. J. R. Stat. Soc. B, 5661))211-277 1994.
We appreciate the insightful discussions with Devavrat Shah. This work was supported in part by the MIT-IBM Watson AI Lab under Agreement No. W1771646, and by AFRL and by the Department of the Air Force Artificial Intelli- gence Accelerator under Cooperative Agreement Number FA8750-19-2-1000 The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Department of the Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
Gneiting, T. and Raftery, A. E. Strictly proper scoring rules, prediction, and estimation. Am. Statist. Assoc., 102 (477):359-378 2007.
Gutmann, M. and Hirayama, J.-i. Bregman divergence as general framework to estimate unnormalized statistical models. In Proc. Conf. Uncertainty Artif. Intell. AUAI Press, 2011.
Gutmann, M. U. and Hyvärinen, A. -oise-contrastive esti- mation of unnormalized statistical models, with applica- tions natural image statisties. J. Mach. Learn. Res., 13 (2), 2012.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Hinton, G. E. Training products of experts by minimiz- ing contrastive divergence. Neural Comput., 14(8):1771- 1800, 2002.
Horn, R. A. and Johnson, C. R. Matrix analysis. Cambridge university press, 2012.
References
Hyvärinen, A. Estimation of non-normalized statistical models by score matching. J. Mach. Learn. Res., 6(4), 2005.
Agarwal, A., Negahban, S., and Wainwright, M. J. Fast global convergence rates of gradient methods for high- dimensional statistical recovery. In Adv. Neural Inf. Proc. Syst. volume 23, 2010.
Hyvärinen, A. Some extensions of score matching. Comput. Stat. Data Anal., 5165):2499-2512,222 2007.
Barthelmé, S. and Chopin, N. The Poisson transform for unnormalised statistical models. Stat. Comput., 25(4): 767-780, 2015.
Jiang, W., Qin, J., Wu, L.., Chen, C., Yang, T., and Zhang, L. Learning unnormalized statistical models via compo- sitional optimization. In Proc. Int. Conf. Mach. Learn., pp. 15105-15124, PMIR, 2023.
Besag, J. Statistical analysis of non-lattice data. J. R. Stat.
Soc. D, 24(3):179-195, 1975.
Lee, H., Pabbaraju, C., Sevekari, A., and Risteski, A. Pitfalls of Gaussians as a noise distribution in NCE. In Int. Conf. Learn. Repr., 2023.
Ceylan, C. and Gutmann, M. U. Conditional Noise- Contrastive Estimation of Unnormalised Models. In Dy, J. and Krause, A. (eds.), Proc. Ins. Conf. Mach. Learn., volume 80 of Proc. Mach. Learn. Research, pp. 726-734, PMLR, 10-15 Jul 2018. URL hwtphhtp/////. /pwppeeedoggs...//// .mlr.nnnn..m........mmmm.......m mlr. press/v80/ ceylan18a, 8aa html.
Liu, B., Rosenfeld, E., Ravikumar, P., and Risteski, A. Ana- lyzing and improving the optimization landscape of noise- contrastive estimation. arXiv preprint arXiv:2110.11271, 2021.
Chehab, O., Gramfort, A., and Hyvärinen, A. The optimal noise noise-contrastive learning is not what you think. In Proc. Conf. Uncertainty Artif. Intell., pp. 307-316. PMLR, 2022.
Liu, S., Kanamori, T., and Williams,
D. J. Estimating density models with truncation boundaries using score matching. J. Much. Learn Res., (3(18)):1336 2022.
Negahban, S. N., Ravikumar, P., Wainwright, M. J., and Yu, B. A Unified Framework for High-Dimensional Analysis M EEttimattos with Decomposable Regularizers. Stat. Sci., 27(4):538 - 557, 2012. doi: 10. 10.1214/12-STS400. URL https:://////. doi. org/10. 1214 12- STS400.
Chehab, O., Hyvarinen, A., and Risteski, A. Provable ben- cfits of annealing for estimating normalizing constants: Importance sampling, noise-contrastive estimation, and beyond. In Adv. Neural Inf. Proc. Syst., volume 36, 2023.
Fisher, R. A. On the mathematical foundations of theoretical statistics. Phil. Trans. R. Soc. A, (2(594-604):309-366 1922.
Pabbaraju, C., Rohatgi, D., Sevekari, A. P., Lee, H., Moitra, A., and Risteski, A. Provable benefits of score matching. In Ady. Neural Inf. Proc. Syst, volume 36, 2023.
10

---

A Unified View on Learning Unnormalized Distributions via Notse-Contrastive Estimation
Pihlaja, M., Gutmann, M., and Hyvärinen, A. A family of computationally efficient and simple estimators for normalized statistical models. In Proc. Conf. Uncertainty Artif. Intell., pp. 442 449. AUAI Press, 2010.
Vuftray, M., Misra, S., and Lokhov, A. Y. Efficient learning of discrete graphical models. J. Stat. Mech., 2021(12): 124017, December 2021. ISSN 1742-5468, doi: 10.1088/ 1742-5468/ac3ace
Ren, C. Misra, S., Vuffray, M., and Lokhov, A. Y. Learn- ing Continuous Exponential Families Beyond Gaussian. arXiv, February 2021.
Riou- ShruuDDrand, L. and Chopin, N. Noise contrastive esti- mation: Asymptotic properties, formal comparison with MC-MLE. Electron. J. Stat., 12(2):3473-3518, 2018.
Shah, A., Shah, D., and Womell, G. On learning tinuous pairwise Markov random fields. In Int. Conf. Artif. Int. Statist., volume 130 of Proceedings of Ma- chine Learning Research, pp. 1153-1161. PMLR, 13- 15 Apr 202 la URL https: http://pooceeeenmmiiieee mllr. press/v1 wwt.0//sshhaaa..aa html.
Shah, A., Shan, D., and Wornell, G. W. A com- putationally efficient method for learning exponential family distributions. In Adv. Neural Inf. Proc. Syst., nett/frumt?
2021b. URL
id-B9WXduMZBEM.
Shah, A., Shah, D., and Wornell, G. W. On computation- ally efficient learning of exponential family distributions. arXiv preprint arXiv:2309.066 0641,, 2023.
Song, Y. and Kingma, D. P. How train your energy-based models. arXiv preprint wrXiv:2101.03288 2021.
Song. Y., Garg, S., Shi, J. and Ermon, S. Sliced score matching: A scalable approach to density and score timation. In Proc. Conf. Uncertainty Artif. Intell., pp. 574 584. PMLR, 2020.
Sugiyama, M., Suzuki, T., Nakajima, S., Kashima, H., Bünau, P., and Kawanabe, M. Direct importance estima- tion for covariate shift adaptation. Ann. Inst. Stat. Math., (0(4):699-746, 2008
Sugiyama, M., Suzuki, T., and Kanamori, T. Density-ratio matching under the Bregman divergence: a unified frame- work of density-ratio estimation. Ann. Inst. Stat. Math., 6445):1009-1044, 2012.
Uehara, M., Matsuda, T., and Komaki, F. Analysis of noise contrastive estimation from the perspective of asymptotic variance. arXiv preprint arXiv: 2018.
Van der Vaart, A. W. Asymptotic statistics, volume 3. Cam- bridge university press, 2000.
Vuffray, M., Misra, S., Lokhov, A., and Chertkov, M. Inter- action screening: Efficient and sample-optimal learning of Ising models. Adv. Neural Inf. Proc. Syst., vol- ume 29, 2016.

---

A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation
Appendix
Glossary
12
Basic Properties
12
B.1
f-NCE
13
B.1.1
Invariance
13
B.1.2
Derivatives
13
B.2
a-CentNCE...
14
B.2.1
Derivatives
14
B.2.2
An Alternative Interpretation of GlobalGISO
14
B.2.3
Proof of Theorem 3.1
15
B.2.4
Proof of Theorem 3.2
16
B.3
f-CondNCH.
17
B.3.1
Derivatives
17
B.3.2
Proof of Theorem 3.4
18
C Asymptotic Guarantees
19
C.1 f-NCE...
19
C.2 a-CentNCE
19
C.3 f-CondNCE
19
D Finite-Sample Guarantees
20
D.I f-NCE
20
D.2 a-CentNCE
23
D.3
f-CondNCE
26
Local NCE for Node-Wise Wise-Spare MRRs
29
Optimization Complexity
30
F.I Convexity
30
F.2 Smoothness
30
G Experiments
31
A. Glossary
For reference, we provide summary of notations in Table 4.
B. Basic Properties
In what follows, Fuler's notation and Lagrange's notation for derivatives. First, remark the derivatives of the Bregman divergence with respect the second argument:
(z( = f(x) f(v) f'()( - y). 0,
(2, v) = (y -x))
0 = --" + "f" (v) xf"'(y)
0y y (2. = (v)
Further, since consider exponential family distributions, have
ds Parh and 00.0, Popisbj.
Lemma B.I. For a three-times differentiable function f. let gy(p) = -pps" (p) + (")
0o.8, A pe)) = <uri,po(ppo + f" (p) -p") + pm (po)}
=i:Wspo(po({" g)po)) ) p² g)pp))).
12

---

A Unified View on Learning Unnormalized Distributions via Notse-Contrastive Estimation
Table 4. Summary of notations.
Notation
Definition
Description
X
IRd
domain of
θ
C IRP
domain of 0
Po (ar)
du(())
(scaled) density ratio
uqa(a)
(1 2²)
h(2) - h(2²) V=h(z))(222 -2-
Bregman divergence of h: R² + R
07 9n)
E arg min Lj (09; qd 9n)
f-NCE estimator (population)
aam (qd. 9n)
E arg min La (0o: 9d 9n)
a-CentNCE estimator (population)
le@
geand (gd: (gd: IT)
E arg min Li (#0: (100 qd. x) AE
f-CondNCE estimator (population)
(ác (n)
E arg min ( Qds (n)
f-NCE estimator (empirical)
Beant (Āa. 9n)
E arg min Lee (de: qd Qn) BE
a-CentNCE estimator (empirical)
geand @d. #)
E arg min f (Øs; Qd. ft)
f-CondNCE estimator (empirical)
R(-)
norm over θ
R R-)-)
dual norm over O
Pmin
minimum density ratio
Pmax
maximum density ratio
B.1. f-NCE
Recall
Cy (0) Ldo (n) = - EE, ( + By (po) - [pe)].
B. 1.1. INVARIANCE
We define an equivalent class of generator functions f that yield the same NCE objective. For a function for let J FFice(fo) & {f: Lace C J. where the notation denotes that the two objective functions are equivalent up to constants, i.e., there exist Á, B R such that Lyee (ou; qd, 9n) AL (do; qd: (n) + B.
Lemma B.2. I (J)) (p as()) bp c) E Juce (Jo) for any a, C. R.
B. DERIVATIVES
Lemma B.3 (NCE: derivatives).
V VeLr Ance (0) Eq. i-po- 1- (po Vo log pol + Eg, f (po) (Pooo/oool po].
V (0) = - Ea I( ((-pef" (po) p8. (po) po V log ps - ppe p (ps) V log pa]
v
+ E4,, (22p) (pe) + pà Tr (po) V P8 log - p (pe V log pel.
In particular, we have
V VoL (0 TEl V o Lipe (0*)] 0.
VOLTE( = Eaga [po- (po. Vo log po- VJ log po.
For an exponential model (b = exp (0. (x)))) we have
V V (0) = Egg ( (po)] + Eg We S.e, (po))
13

---

A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation
V2l (0) L T (.ce. + L S.ce...n (po)],
where
f. = -pf
c(1)
S.ce.f.n (p) = p² (p),
s (p) = pg.(p),
5 f.ce (p) = p² - g)(
In particular, if qa())
to (x) for some 0r
VoLT (0*) = 16[ V
Ence (0)) = 0,
Vi2T (0*) 16 V V (0*) - Eg - (vrb f(o )1 Ea.lvb T
pa- (pe-) )}
B.2. a-CentNCE Recall that
rx(x))
(x) = (E.g. (r0(()))) (x)))
and
(0) A (0:
Qd, (0: (n) ) A
Egg [ro- (I:] [x:]
- Egg 1r (F.) ( [r]())))* 1="
B.2.1. DERIVATIVES It is easy to check that Lemma B.4.
V log Pojo = 2b - Eg p
V log Farm aa Egg (Rlrgh TeotTTogr. rg;aa E.g. [ybr 8:o Egg (1 }
B.2.2. AN ALTERNATIVE INTERPRETATION OF GLOBALGISO
Consider unnormalized model (dv(x)): : O). For data distribution ga(zr), which we have sample access, assume that there exists 0* θ such that (2-(2) ga (z). Let gn (x) be reference distribution which makes Eg, [log )e (x)] exist for any 0 E ④. We define a "centered" unnormalized model
L (x
(1()) A
Lo Dog (=(=]]
and denote its partition function 2() (z) da. We remark that
Fyg [log de (xr) = Eyg, [log (x())) - Ey [log che (x)) = 0.
(10)
We then define an objective for distribution learning
In (x( Egy
Lgiso (0)
do (x)
If (g(x)) = eXp((0, vxxx)) is an exponential family distribution over a compact support X' and (x(x)) is the uniform distribution over X,, then it boils down to the objective function studied by (Shah al., 2021b).
14

---

A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation
Fisher Consistency To understand the property of the objective, introduce another unnormalized model
00 (ar) _ ΦB (x)
On (n()),
60 (x)
and denote its partition function and the normalized distribution
5o, (x) dr and (0) - S0,,02 (x) Z(01: 02)
2(01,02) -
We can then show that Theorem B.1.
log Liiio (0) Digallgo ,) log 7(0*).
As an immediate corollary, we can prove the Fisher consistency of the objective function.
Corollary B.1 (Fisher consistency). Let 6 E arg ming (0). Then, Φo- (z) o Ba(z) E supp(gn)
The proof of Theorem B.1 readily follows from the following lemmas. Lemma B.S.
Cgiso Z(0
2(0-)
Proof. Consider
Lgsso (0) A (a(x)) (n(x) da
(x(
/ 3o (x)
gn (x( dr
(00) de (xx
/
Φ0 (x)
On (x) dx
7(0-)
Φ0(x)
7(0", 0)
2((0))
Lemma B.6. For any 01,02 O2 E θ,
D(qnll90,,0.) = log Z(0,,0,)
Proof. Consider
D(gallae,.e)) = Egg log (x)
Q0, 0g (x:)
= Eyy [og Z(01, 02) + log Φo (x)
Φe (x)
log Z(0,, 02).
Here, in the last equality, we use the fact that log (x is centered under gn (2), as alluded to earlier in Eq. (10),
B.2.3. PROOF OF THEOREM 3.1
Theorem 3.1 (a-CentNCE subsumes MLE and GlobalGISO). The following holds:
(a = 0: GlobalGISO) For an exponential family (z) X is bounded and 9n (x) is uniform distribution over X, the O-CenINCE objective Co (0; qas 9n) is equivalent to GlobalGISO (Shah et al., 202 lb).
15

---

A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation
(ar 1: MLE) If Z1 (0) is assumed to be computable for each the I-CenINCE objective (1; úd 9n) is equivalent MIEE (Fisher, 1922).
(a 1: MC-MLE) If (0) = E. [ estimated with empirical noise distribution 9a (2), the I-CentNCE objective (8; qa. Qn) is equivalent to MC-MLE (Geyer, 1994).
Proof. When a + 1, the centering becomes the standard normalization, i.e.,
@ 2z)
2 (I)
0 (z)
lim
(B
)a))1/aa
2 (x))
de
and thus the objective becomes equivalent to the MC-MLE objectives:
Ê (0; qd, qn) - ( log
Eyu(()
log 1 + log Egg
Φ0,1 (x:)
tha (x:)
In (n(xr)
When Z1 (1)) Eg
( (n(x))
Z()) is assumed to be computable, this becomes equivalent to MLE.
When a 0, the centering becomes
Φ0:((x) lim Φ1(x)
Ø (x)
(LEn, I( ( jaj2z/a Boncer (log
(a(r)) valay
and the objective becomes
(n(2))
(x)
Co qd, 9n) 1gg(()
Bx(((z)
(e) Dog mn(=)
(11)
(0.0
1 (xx
In particular, for the exponential family, we have
(w) log @0(x))
log (0) A Fy
(0, (22)) E. (a) (lg
(x)
where Ji (x))), and thus the objective becomes
(o; ga. (n) = (wy (gn cxp((() (bx) v5)))]
modulo additive and multiplicative constants. When the underlying domain X' is assumed to be bounded, we can set gn (x) as the uniform distribution over X'. In this case, the NCE objective boils down to the global generalized interactive screening objective (GlobalGISO) studied by Shah et al. (2021b).
To provide a comprehensive view, we summarize the connections in terms of the objective functions that correspond to the unified estimators Table
B.2.4. PROOF OF THEOREM 3.2
Theorem 3.2 (fa-NCE and a-CentNCE estimators are equivalent). For a A θ x I in the augmented parameter space, let Ale & [0: (8,v) e for some R} denote the subset corresponding e. Then,
arg min
CTe (nce qd. 9n)
arg min Leent(0; qd. 9n).
Proof. On one hand, we first note that + Lace (0; Gd. (n) is convex, and for each B, the minimizer (0) of the centered objective satisfies
Eq, Ir
(0)
Esg [r
16

---

A Unified View on Learning Unnormalized Distributions via Notse-Contrastive Estimation
Table 5. Existing estimators as special instances of NCE estimators.
Existing estimators
Corresponding NCE objective
MLE (Fisher, 1922)
C (0; qd- 4n)
GlobalGISO (Shah et al., 2023)
C5 (0; Qd Un
MC-MLE (Geyer, 1994; Jiang et al., 2023)
L (0; Qd 9n)
IS (Pihlaja et al., 2010; Riou-Durand & Chopin, 2018)
hh (0: da. Qn)
eNCE (Liu et al., 2021)
y (0; qa, 9n)
Pseudo likelihood (Besag, 1975)
C (0; qd, Qn) (local)
GISO (Vuffray et al., 2016; 2021). ISODUS (Ren et al., 2021)
C (0; 9d 9n) (local)
Moreover,
V VoLTo Lace (0: qd. (n)
c2la-11 (ra (1la1)/ logrol E [ră Vo logro:
so that the fa-NCE estimator f._ (Âa. Gn)
nce
(0) (Gd. dn), 1/7000 (Ād. (n)) satisfies
E 1r V log rel Beg 1 B.g. ( F Vo logrø]-
(12)
On the other hand, we have
VoLce (0: Gd. gn) = Egg. lrg Son-'ve 'vo Jogro](EB) (E) (r81) - + -Egb
'1 E (r81)
1-2
Egg Vo logrol.
which implies that the a-CentNCE estimator geent (qd. 9n) is also root of Eq. (12). This establishes the desired equivalence.
B.3. f-CondNCE
B.3.1. DERIVATIVES
We first note that Lemma B.7.
Vro(z, = r0( y) V logro(z,
1
Vr (I, = ( v) Vrg(z, y) = ro(x,y) v) Ve (x,
In particular, for an exponential family distribution on = exp((0, vxx))) ), we have
Vo log (o( y) = (xx) (v)).
V log (o( 0.
Lemma B.8 (Conditional NCE: derivatives). Let (r( v) pa for shorthand.
V Loond (0) E. Ee
P9 fd (p V log Po),
v Vg (0) = [e (-pppppppp'p__ 0 1r (po) V log pa VJ log Pe (pe)" log pa
+2pp 8 (p + PT fr (p) ) log pa VJ log PT" 2 (' ( log po).
For exponential family distribution (x exp (z)))) we have
V WeLdond(() (0) = (enly))(s (
L (po(s,, y))}
2 2roond (0) = ( (v)))(((x)) - (T (po w))l,
17

---

A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation
where
Scond, _ ( = = _ + Since scond, x(p) + -g((p)) = -Snce, Sace, + Snce. S.ne.
In particular,
V of (6*) = 0, V Vg (f*) (w) (WV(() +v)))((())) ((((f)))-(())))) pa."
B.3.2. PROOF OF THEOREM 3.4
Theorem 3.4 (Asymptotic behavior of empirical f-CondNCE for small e). The empirical f-CondNCE objective can be written as
Capnd (00: (o. =-((1
+
V log Go Tulee (1) (Øu; qd, à + o
Here, we define the empirical sliced SM (SSM) objective (Song et al., 2020)
C (do: Go 4s)
A (x)((()))) 0 TV log (x()) + (( V. log oz)))
Proof.
Let (O, a Zgndddoo da, (.). Note that
( c) = F - f(r
where we set r have
de 6o(=+ce)
shorthand notation. Since by chain rule have à de logr V log Φ0(x + TV)Tu,
(0,cc) = -Eas())( Vx log di (o + Tu (r) Tv)Tu(()f(())
+ fast 1" {
dc
and
de² 0 (0,cc) -Es(a)(())) n. V log (o (0) 1 (+))
82
log (o (rf + (r)
2
5m (-))
- 5 ! [
,2
Hence,
(0, c)
=-((1),
c=0
0
Cy e)
2f (L) V log do (z)
de
c=0
8²
C, 6)
(2 V do (V (x)
de2
=00
(1) (ds: qa. 4s).
Plugging these the second-order Taylor approximation of + C,(0,e) around 0, i.e.
(0,,, = C, e)
+ a c, e)
0
é (0, <)
6² + ole2),
0e
2
concludes the proof.
18

---

A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation
C. Asymptotic Guarantees
We can establish the asymptotic consistency and normality the estimators. Though we present the results for exponential family models for simplicity, one can derive the asymptotic covariances for general unnormalized models and generalize the results. All the proof' are straightforward from the application of standard M-estimation theory, see, e.g., (Van der Vaart, 2000), so omit the proofs.
C.1. S-NCE
nce Theorem C.1 (f-NCE: asymptotic guarantee). Let fimamn.
(@nce fina.ma"
ance be a solution of
Fina,
nce ffmam arg min DEEXR
C (
Let nn A And for some B > O. If (T) & (() nid - uniformly over B θ R. finamt
2 (8* as
nd + xc. Further, O E int(@), we have Vna fing.mm - 0*) 4 NO, Vicc), where we define Viee a I C
I Egg lpo (po-)
mM (1 3P0r Pa. Parf" f" (pu.) )Bbb - -1+ ) lpo ) f" (pw ) B Ipl f" (pu- )oh]
C
for (bx) [+(x);;JJ E 1BP+1 provided that I is invertible. In particular, the asymptotic covariance Vf satisfies fing is PSD matrix, for any f.
yirce 2 fiog" or equivalently Vi
This result has been known, but we present rephrased version here to contextualize our contribution. The asymptotic convergence beyond exponential family was established in (Gutmann Hyvärinen, 2012) for fiog-NCE and (Pihlaja et al., 2010; Uehara al., 2018) for f-NCE. The optimality of flog established in (Uehara et al., 2018). It was independently proved that the original flog-NCH NCE estimator asymptotic covariance not larger than that of the fi-NCE estimator (and thus the MC-MLE estimator), which they call the IS estimator, in Loewner order (Riou-Durand & Chopin, 2018), In the same paper, the asymptotic guarantee for the Jiog-NCE and IS estimators was shown for a general unnormalized distribution under a non-i.id. setting in (Barthelmé & Chopin, 2015).
C.2. a--CentNCE
Theorem C.2 (CentNCE: asymptotic guarantee). Assume that any expectation over qa in the a-CentNCE objective can be computed for any without samples from qn. Let (ceent be a solution of
acna
jeent arg min Ceent (f: Go: 9n) BEB
If Leent (0; qd. Qn) 2s Leent( (0; qd, (n) as na uniformly over 0 E θ geent 1s 0 as ai oo. Further, 0 iin(()). we have (Ta(( (gcent asra
0+) $ NO, )rent), where we define Veent & -
Ia (1 Bm [F ( Eg, 188 v1) Eg [i 1a []
+ +BB 1 dd (B aa Teb - Egg 178 BB 18 b)
Ea A Kgg (2200-1))) (w) - E.g. Fo (16 E.g. [0 -2,)),) T),
provided that To is invertible. Here, note that 78 (x) E
mi
In particular, this result recovers the asymptotic convergence of MLE for a = 1, and generalizes the analysis of GlobalGISO of (Shah et al., 2023) beyond when gn is the uniform distribution.
C.3. CCCondNCE
Theorem C.3 (ʃ C/-CondNCE: asymptotic guarantee). Let
be a solution of Jind
E arg min frond (8).
BE0
19

---

A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation
If (0) & (0) as nd + uniformly over θ grond E 0 as ny + co. Further, if 0 int(()), we have
/nn(fffnn fine -0)) 9 NO, Deand), where we define Visand & 1 T
f A (ul())x((l=)) 8 ( (v(x)) - (()))(b(a) (()))T]
č A (ull)) b (po. (bb(1)) - v(0))(((a) v(1)'
provided that y is invertible. Here, pa - pa (r, y) and Scond. (p) - p "- +))))))
D. Finite-Sample Guarantees
For the finite-sample analysis of the regularized NCE estimators, we invoke the result of Negahban et al. (2012):
Theorem 4.4 (Negahban et al., 2012, Corollary 1). Let .1..... 2N be i.i.d. samples drawn from a distribution p(z). Let ho(2) be convex and differentiable function parameterized by 0 E Q. Let L. n Ei= ho(zs) denote the empirical objective functionn. Define
On arg min argmin
( + (0) (0)
where An is regularization penalty and R: θ Ry0 is over Q Let 0r arg Ep(=) (ho))] Assume that
1. The regularization penalty >, satisfies An 2 2R (Vo((v))()"),, where R" O² + Rzo a dual norm of R over the dual space e"
2. The empirical objective (0) satisfies a restricted strong convexity condition at 0* with curvature i.e., ( (0,00 0r) xil0 0 0-
Then, the estimator 0, in Eq. (9) satisfies
1|0, 0¹112 S 7R,2
D.1. F-NCE
Theorem 4.1 (f-NCE: finite-sample guarantee). Pick strictly convex function IR, - R. Define
(pmins Pmax)) & (Le inf
Pe(x),
sup
pg(x))
=EX,
2EX,
and define the quantities in Eg. (5) accordingly. For E (d, n). define
A Amin (B.g (1)1bT)).
min,r
Let @nce, R be such that
{ cgp Qn) + An R(0}
E arg min BE0
for some Xnam, > 0. Then, for any A > 0 and 8 E (0, 1), there exists a choice of Xna, such that 0
Jmmam - -0 5
with probability 21 1 - 5, provided that for each E (d.n).
(B)1) nir, frr)
2 2
1/.2
TR: 7 Vixxx
n, = max
2
nce nnce,sd min,d
(2)
+ (2)
nnee, f. min,n )2²
A nce
log p²
nAnce )
min,r
20

---

A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation
We need to show two properties. First, the empirical gradient VoL nearly zero 0 0* with high probability (Proposition D.1). Second, the empirical Hessian ViCte (0) has strictly positive curvature (i.e., exhibiting restricted strong convexity) at = 0* with high probability (Proposition D.2).
Proposition D.1 (Vanishing gradient). (ef. (Shah al., 202lb, Proposition F.I).) Assume Assumption 4.1. For any ò E (0, 1), <>
Vo lmmax Se
2 (B a.d
with probability 2
5, if,
log af for each {d,
Proof. Recall from Lemma B.3 that
VoLice(0) - (" + [o,
V
Therefore, we have
E E[0g Lr (6*)] 0o, L7
Since (x)), S.a (pa(())) S tm B fid and (x)(r) ( (1) union bound, we have
S.d (po(x))] S tm B
Bece by Hoeffding's 's inequality and
P( 0o 2. ())((0)) 1 < 2exp
ndd
+ 2 2exp
Un
= 6,
20 max fid )2
2162 max
(B
f.e,
Dd 2 20l (B ) g²
log 3/ and fi 2 24 (B finl 12
log 3. By taking a union bound over different coordinates of 0, we
conclude the proof.
Lemma D.1. (ef. (Shuh al., 2021a, Lemma E.1)) Assume Assumption 4.1. Let be either qa qn. For any > 0,
max EEp [bb E, [bvabbsl $ E2.
with probability 2 1 82. if
2 1
2p² 82
n2 2 max log
2
Proof. Since (x)) s 2/2 waa is a bounded random variable, by Hoeffding's inequality, we have
E4 [bb0))]] <2)
2exp
2
Taking a union bound over j E [p) leads to the desired bound.
Recall that for function h: θ + R, the Bregman divergence is defined as
(0,00) A h(0) ((.)) - (Voh(0.), 0.)-
Proposition D.2 (Restricted strong convexity). (ef. (Shah et al., 202 Proposition E.1) Under Assumption 4.1,
(2)
Enpa (8, 0*) > 1
f.ce
Amin, + (22) n.e, deuin.)) 1) 801
with probability 2 1 - 6, if >
8 max Aminn
20² log
3
for each E n).
21

---

A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation
Proof. By the intermediate value theorem, there exists & E (10 (1 1)0": E [0. 1]} such that
Egg (0, 0-) = 27 (0) - - ( (0*), 0*)
7 (0- 0.) - (V
8*).
Here, note that & depends on Os and gn. Let 2 (1(x)), 0 8*).
(0 (- 0*) = - 2 /22 (pg)(Pp)) + L= pf - 96 (pE))
y
b ((2)
f.ce Ee [2-] + b (2)
f Bun 1=21
1
(2)
= (0 - --) n.ceffdd
Eg. [+> + n B. (o (0 8*).
We can lower bound the quadratic form as follows. The first term can be lower bounded as
( )² "Bg j [(0 8²)
= (0- - T [ Eg [ + Egg (nbert - 8r)
= Σ (0- - 10-0) ( [Wbbba) E (wbe)))) - )r)) + (0 -0-0 -0*)TRR, [1) 07))
2 Σ 10: 0-;1 ;1- [ - Werebll 100 - 00 + Amin. 1 - 00
15
-2210 - 10 + Amin. - 0-11
22 01 + Amin, 1 0/
al - 1-
with probability 8' if nd
2
log 2p² with e2
Amind 2
Here, we apply Lemma D.1 in (a), and use the definition
of 7122 to bound |0 1 0 1| 71 ||0 11 in (b).
Hence, by a union bound with 8' = 8/2, with probability 2 1 - 8, we have
(2)
Ence (0,
f.ce
Amin. bee, fin Amin.o) ||e -
1
if rid
83 Vine Xmtund
log 1p and min
83 max Amina
log
10² 0
Proof of Theorem 4.1. First, note that
R (V S 70 VoLc (00) Imax
by definition of 7 Then, by Proposition D.1, we have (oo |mna Se with probability 2 1 81, if
nr n.ce/.1)
(( 2
12
2p
log
2²
for each r E n). Given that this event occurs, R* R* (VoLTe((0*)) & YR-v00E, and thus we set An + 200 to satisfy the first condition in Theorem 4.4.
Now, given An. 2R' 2R (V (Negahban al., 2012, Lemma 1) implies that R(0RER 00)) < 4R((0), i.e. Gece, 8" 40. Then, by Proposition D.2, have
Aoc (0,0*)
2 kl|0 0 1
22

---

A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation
with probability 2 1 - 52 if n,
8 mina
log
4p for each E {d, n). where
1
f.de min.d + In f.nin.)
4
Now, by taking union bound with 81 = 62 = =./2, with probability 2 1 - ò, we have
PR
6 YR:22
|10 0- 5
K
K
with
AN 00 78.
provided that
nr 2 max
72 (1)
f.re
:00 ) Ti
2
Vixxx
log
4p 87
max
log
8p²
8
X2 min,
1152( fir 1R, 7 :00 Waaxx
(1)
4p
8 2 max
8p²
=max
log
log 8 S
A2 16)2)
nce, 1.d n. minud Unce,s
Iicessnn Amin,n) )2
=m(max{
(L (nnce,sr) ),,)) 7R R
222 :00
max
11:2
max
p² log
(2 nce, Amin, + 612)
ncce Amin. )2
min,
8
for each E nd.
D.2. a-CentNCE
Lemma D.2 (a-CentNCE: derivatives).
VoLo (0) = -- 1all--a-1V1 V log [o:a]:
VgL. (0) BEag [:::0 ((1 Vo Foga Vo log V8 1ogogii Fo:a)]
Define
Ca.ns Cov( Vo
for nid 2 1. Then, Co.ny Co for any na 2 1, where
C, A Eng P0
12aa-1) log Fo Va log log
We also define
Ig a (3C)(0") = Eg 1o ((1 a Vogiog V logig.
V2 1og
Proof. From Lemma B.4, we have
Oo (0)
(1 xx) Ege. [3o, Fo:-!
1
-Bag 00:0 Os, Foaal
-agg 1-0--1 (s/s - En (o
From this derivative expression, the computation is straightforward. Corollary D.1 (GISO: derivatives).
VoLo( Fng ( (x)D),
ViCo Egg |o ( Eg(()))(* IE).))]
C, Eng 1 (1b (xl - E,,D
I Eyg 11 (2 (vg F.(DD)..
23

---

A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation
Proof. From Proposition D.2,
VoCo(0) B log Te,0],
(2C)(0) E. f V log Vo log 7 V log fa:0)s,
Co Bu F V lo Vo log 1
To B 1 V log f Vo log log - v log o)]
Since
fo:u exp(10, v Bbgl1b)) E.
((x)
V log Fe: = Eg [10),
V log O,
the quantities can be further simplified as stated.
Corollary D.2 (MLE: derivatives).
VoC( E - V log pal:
V)L((0) E log pol:
G g 9 lo- Pa- V log p
Σ Eggl - V log p
Theorem 4.2 (a-CentNCE: simite-sample guarantee). Pick a E R. Define
(Puins Praux) A ( inf
(o)a sup zEX,
po:a (1))
a
and define the quantities in Eq. for fa accordingly. Let Po-ya (x)
and let
Egg (( yoj
cent A Amin [(b p0 Eg (o-
cent minn,
A Amin (Beg. [1by/T po-;al E.g. (a P0 TT)
Let (cent.R. be such that
gcent, E arg min ( [cent (f; qd, 9n) Am, R(6) 8E8
ornng
for some Ang > Define Vmax, V/max + Bag. : mmax- Then, for any 0 and 5 E (0, 1), there exists choice of Xan such that 10
0+ 1|2
A with probability
provided that
(B nce, fo.d ) 22 :00
1/2 max.a
nd
max
(6 nce, )2((1 Acent min. cent 12
M 24 max.o
log 11 8
()cent
Proposition D.3 (Vanishing gradient). (ef. (Shah et al., 202lb, Proposition F.I).) Assume Assumption 4.1. For any E (0, 1), e>0,
(f*) Sc
2r 2(n-1) (Vime + [| (oPg. mlmma)) log 3
with probability 2 1 5, na
24

---

A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation
Proof. Recall from Lemma D.2 that
V (oLa(0) = Egll-r8;00 Vo log Fo:al = -Eg, (3 + Eg F0:oo |Eg [xiT
0:a.)
and it is easy to check that
E B ((VoL((0)) =Vo)==oaa(())) -R 00 Pugo 10 Bg (1/0 8r;a 0.
g, (- 1)1 S minua (1/max + ||B (vb 0 |mxx), by Hoeffding's inequality, we have Since (10
ndd
P(I 0o (0)) 2 c) S 2exp
2r 2ia-1) (1/max + [|Bn ( :
(Vmmoe + (V le m.lllmae))
if nd 2 proof.
log 2 8. By taking union bound over p different coordinates of 0, we conclude the
Proposition D.4 (Restricted strong convexity). (cf. (Shah et al., 2021a, Proposition E.1)) Under Assumption 4.1, we have
E.. (0,0*) 2 maxx (2(1 ax) min,d + O }11 100 0- 12.
with probability 2 1 5, nf (
||E /- an mmx) )a
log
2p² 8
(X a
Proof. By the intermediate value theorem, there exists & E {to + (1 t)0 :t E [0. 1}} such that
A E. (0,00 = i.. (0) - L. (0*) - (VoLa (0*), 0r) (0- 8 V 8¹).
Eg ( [8..] for shorthand notation. Here, note that & depends ğa. Recall from Define 36 Ez [obig Pead and T
Lemma D.2 that
ViLa((0) 1: ((1 Vo Fe: V log V log Fo:a)
--T (1 Ba [= (1p 3)) 3)J + alE (o 1 (1iot -//0/ ).
Let A (V(x), 0*)
(0- (0--0)TT T (. (E) 0*)
(1 a Bi t: ((0 0*) )T) [(b: - - + aT 8:a [0
(')T(
--m w/b )(0 - 0*)
2 (1 - a) ) max. E [(10 0') (v- - 5)) + a - Acmm 10 0+
- a: {11 (0 Eault 1( 3) - 3) 00) + a cant
min,n 110 - 0-1133.
We can lower bound the first term as follows.
(4)
(0 8-) TT, ((0b - V)) - T) 0*)
2
02 101 + cent |0 0²
(6)
2, 110 0*1/2 11 + A | - 0*1
Amm.al
al
00
25

---

A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation
with probability 1 8 if na
2(Vmmar + [y unl mmxx))
log
2pth with E2
cent
Here, we apply Hoeffdiig's inequality
of similar to Lemma D.I in (a), and use the definition of 71.2 bound |10 0* S 71:2 0*112 in (b). Hence, with
2
probability 2 6, we have
E (0,0*) 2
pa-1
{
a + axAmin 110 0 11:.
max,o
provided that ni 2
8y eaaaa + an (18 max)
log 20²
6-
Proof of Theorem 4.2. First, note that
R R'(VoLa(O')) S PR-:00 || [V (8") max
by definition of yR Then, by Proposition D.3, we have || ( mmmxx Se 6 with probability 2 1 61,
nd 2 min.a (V'ax
2r 2(a-1) 1)
[gg ga
max))
2p
log 81
?
Given that this event occurs, R (Vo & YR and thus we set An f- oo0 to satisfy the first condition in Theorem 4.4.
Now, given An. 2 2R" 2R (VoL,, (Negahban al., 2012, Lemma 1) implies that R R.jsent. 0*) 4R(0"). i.e.
geent. 8" E 46). Then, by Proposition D.2, we have a,ns
cpp (0,0") k:10 0*
with probability 21 1 52 nd 2
87 (1 + E (e mllma)) )
2p² log
where
(Amin, .²²
5
1 {2(1 a)mmm.
pa-1 a
admimn
pa-1 max,oo ((1( a)) )
cenn
Now, by taking union bound with 81 62 = -/22, with probability 2 1- ô, we have
3R:2
6 A
||0 0" 1| -
with
Ax Gig 72:2"
provided that
72r
2(a-1) 2 min,a (1/max + E. (wrr :a max /2 7 * 2p 87 (Vmax + 1|1 lybra |
1p2
n, 2 max
log
log
(Acent )2
1152r 1 2(a-1)
min,a (Wmax + E,g. ppb 0- and max P7 Tie :00
2p 87/.2(Wmaa + Egg. Px 0-:a Imaax) ) (Acent
4p²
mmxx
log 8
log
2(2-(11)) 2(a-1) 1)
m [(1 a cont a cent 12
2(a-1) 1) minna
(mmxx + || (ebigg : max ) 7 2 (Vmmxx + || ( a (allmml) mmmx))
p2 log
= -
2a-1)) 1) A 2 max,a
{(1 a Aeent a Aceeett
min,n }2
(Amat, )2
g
D.3. /-CondNCE
Theorem 4.3 (f-CondNCE: finite-sample guarantee). Pick strictly convex function f: R4, - R. Define
Pinin A
(..))eeupp(o@@@)))le))e.. Po y).
inf
26

---

A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation
Piax
sup (=N))lupple(@@)rtlll)...e.ee
(o
and define the quantities in Eg. (5) accordingly. Let
cond
Amin (Banae - v()))))
Let goond.R be such that y,ne
goond E arg min (gomd (0; dd, (t Ana R(0)} 0EG
for some Ang O. Then, for any A > 0 and (0,1). 1), there exists choice of Xn such that fins - 0- = A with
probability 2 6, provided that
(B() fond
B())
fond.f.n 12 TR 1 100
maax
7ld = 2 max
cond. + b condd.f. ) (Arond )2
log n1²
(Acond) 2
8
Here, bcand, and B Beodd, place of (pmin: Pmax))
ford.ffr are defined similar to Eq. (5), where the infimum and supremum are taken over
(Pmin
Prax
in
Prnax
Pmitn
Proposition D.5 (Vanishing gradient). (cf. (Shah al., 2021b, Proposition F.I))) Assume Assumption 4.1. For any ò E (0, 1), e
|1 Cood (8*)
max Se
Sola mase
(1()
B r.
with probability 2 1 8, na
log 3
Proof. Recall from Lemma B.8 that
VoL (0) = (ow --()))))ee)e) + c(1) (n
and it easy to check that
E V c Acond
(ooond(d (0*)
Velgond = 0.
Since
((bv()) - (x))))) Snce, s.d ( (x. m)) sn (pg (r.
(1)
$ (():(v) 4/((x) nce, f.a (P (ax v)) + (()(()) oh( (po (2. y))
s 21/max f.nd.f.d
(1(1) ((2) f.nd
by Hoeffding's 's inequality and union bound, we have
P(I 0o, 23 (0*)) 2 e) S 2exp
=
81/2
(B(1)
B ((I)
max fond
f.nd
(v I. B ) log 5. By taking union bound over p different coordinates of 0, we conclude the
if na 2 proof.
Proposition D.6 (Restricted strong convexity). (cf. (Shah et al., 2021a, Proposition E.1)) Under Assumption4.1,
A Exgan 0.0*)
(beond.f.dd (2)
cond, cond 0 0-
with probability 2 1 - 5, if 2
128 4
log 2x
27

---

A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation
Proof. By the intermediate value theorem, there exists & E (10 (1 l)0": E [0. 1]} such that
Exg (0, 0*) = Écond (0) - p (0*) - (o (0*), - 0*)
1 (0 8*) T Lg (8) 6*).
2
Here, note that ç depends on (a())((y))). Let = _ (vb(x) (1y), )²).
(0 - -- (E 0*) = ( [[nce S (p ) + Snce fo (pe =
2 (b Woond.r S Sa cond, /f ) io (z) (()l=))))
ffnd. Sd cond. (0 0* ((((())))))) - (())(((( (()) 8*).
We can lower bound the quadratic form follows. The first term can be lower bounded
8*) (A -8- "0-8""EB "eteaeeetto - (e)) 0*) (a)
2 c2l A 8*117
ccond ||0
(b)
2
-227, 2 ||0 - Amnd |0 0-
cond min
0 0
2
with probability 2 1 - 5 if na 2 324/ max log 5 with 62 271.2
2p²
Aman
Here, we apply Hoeffding's inequality as in Lemma D.1
in (a), and use the definition 71.2 to bound ||0 0- 11 S 7:2 8* 12 in (6).
Hence, with probability 2 1 8, we have
Exund (0,0*)
(1
f.dd + ((2)
fond, f,n / 1 0-
ndd 2 log 2p
128 Paaxx (Avond
Proof of Theorem 4.3. First, note that
R V (0*)) S :R | Lg (0*) |lmm
by definition of yR Then, by Proposition D.5, we have V L (0*) mm S e with probability 2 1 - 111-6,11
S max (B(l)
B f.ond...n )
cond,,
2p
log
2
81
Given that this event occurs, R* (V (zond S oo and thus we set An < 200 to satisfy the first condition in Theorem 4.4.
Now, given An 2R" (V V (())) (Negahban al., 2012, Lemma 1) implies that R(@rand. 0-) S 4RR(8"), i.e.,
geond. 0+ E 40. Then, by Proposition D.6, we have
fim
A Cxme (0, 0-) 2 K 0"
with probability 2 11-60 52 if na
128 a Pmax (Awrd )z
log
2p bg
where
(oondd S.d (12) A
cond
Now, by taking a union bound with 8, = 82 = =//2, with probability 2 1 - ō, we have
3R.2
6o YRR:2 A
||0 8* 2| <
28

---

A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation
with
provided that
7d 2 max
288(B (D
288
cond.r.a + B Beand 3
2 VR- :00 V
max
log
4p
128
1122 1/4
max
log
4p"
A
6"
(Amnd) )2
8
max
1152(B
fond.f.d + B
cond. )
2
TRt :00
log (Acom)) )2
4p
128
11:2
v max
log
4p²
( f.n b ) (Acond) )
(2)
(B
fond, Bean Boodd.fnn 2 RR:2 7 soe w max
7122
n² log
= mmmm max
22 f.d + 11 beand, ) (Acond) )2
(Acond)
8
E. Local NCE for Node- Wise-Sparse MRFs
In this section, we illustrate how one can construct local version of the NCE principles introduced the main text for, e.g., node-wise-sparse Markov random fields (MRFs). The notation herein follows the convention of (Ren et al., 2021) with modification. We use a boldface notation x (z1...., Tp)) E X R for the purpose, and regular-for font variable is assumed be scalar-valued. We assume that the exponential family distribution consider is described as (0(x) ex)(E(x)), where the (negative) energy function is
{xx) 2 Σ (11f((x1),
IEI
where a (sr: I E T) for some I C 2(Pl is collection of basis functions fr: llke 2 - IR, each acting upon subsets of variables Xy. Note that is often called the sufficient statistics of the model.
To deseribe conditional model, for each i [p]. define I, - II e I: i: I}. Then, we have p & se bo( e exp(E:(x)))
where
( A Σ (x)1(x))
IEI.
We remark that the pseudo-likelihood estimator of Besag (1975) is defined
0 ف arg min
log
O2
Po
where 0g is collection of all parameters that affect the node conditional model ( xx)) among all parameters O.
For n-th sample x("), let 2²)") denote the j-th coordinate of model
x (n), To apply the f-NCE principle, define the density ratio
po(a; xxx)) 6 (xx((x))))
vgn (2:2)
for choice of reference distribution gn (x). For each node i E [p], we can derive the local f-NCE objective as Bx 11 (co (do ()): go (z: (x:))]
Banxss))0nxx) Od (2: (x.))
(0 (0(or1/2x)2) - [0(011)x)))]
vgn (cx))
ven (z,)
uggn (2:4)
- If |xx (x(xx)na.)) (p) - fpo(#, (x,x))))).)
(13)
In similar manner, one can derive the local a-CentNCE, which recovers pseudo-likelihood (Besag, 1975) for a = and GISO (Vuffray et al., 2016; 2021: Shah et al., 202 and ISODUS (Ren et al., 2021) for a = 0, respectively. We note that Ren et al. (2021) justified ISODUS only from the stationarity of the objective function at the optimal parameter, while the connection established here between these interactive screening objectives (ISO) (i.e., GISO and ISODUS) to NCE provides a natural theoretical justification.
29

---

A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation
F. Optimization Complexity
F.. Convexity
Proposition F.1 (f-NCE: convexity). Let g)(p) & (p) + f()). If f²) 2 gs(p) 2 O, then - (0) convex. In particular, + (0) is convex flog and fa for [0,
Proof. By Lemma B.3, we have
V V cre (0) Bg [ S.ce, (po)) Egg, l S (.ne)
Hence, if gr (p) 2 O and ("(p) 9و (p) 2 0, then ViLnce (0) is nonnegative combination of two positive definite matrices, and so must be positive semidefinite.
remains to show that the condition holds for all I's in Table 1. For the asymmetric power score fa(p) = p² with < 1, first, is easy to check that p fp( is convex.
(p) p²=
(p(p) -(pSa" (p) (a(P)) (1 a)pa>
Since Ia (p) 2 0 and (p) 90 (p) apa- 2 2 0, 0 - Lnce (0) is convex by Lemma F.1. Note that the same calculation holds for a E 10.
A counter example of convex functions / which do not result in convex objectives is fa(p) for & [0, For S-NCE, while
(p) -p--2 2 0 for any a, 91 (p) = a)p- 2- < 0 for a >> 1 and a * 2 and (p) 9f. (p) apa- < 0 for
a < 0. For a = 2, 95. (p) = - 0.
Proposition F.2 (CentNCE: convexity). For a [0, 1], f + Ceent (0; qd. 9n) is convex.
Proof. For (0, 1), note that write
+ log Ea qd, 9n) = log(1 - a) + log Eg l0gg-lg-000
(x)] +
a
log Eg Jp (x)].
Here, the second and third terms can be understood LogSumExp operations applied on the linear function 0 + log (z)), and the resulting function becomes also convex. For E (0, the MLE and GISO objectives are well-known to be convex.
The proof of the following proposition is similar as above, and we thus omit the proof. Proposition F.3 [f-CondNCE: convexity). +pp===")+(((")))))) 2 then + (rond(() is convex. In particular,
0 Lcond (0) is convex for Slg and for [0,
F.2. Smoothness
Under the boundedness assumption, show that f-NCE objective function is smooth with probability 1. Proposition F.4 (Smoothness). (cf. (Shah et al., 2021b, Proposition B.1).) Assume Assumption I. + Lice (0) is smooth function with smoothness constant
F.ce f.d
((2) f.ce
max
Proof. Recall from Lemma B.3 that
2 VoLre
fnce (0)
Bag l c22)
s.ce,r.a (po)) ) Egg
lyv. c (.))
30

---

A Unified View on Learning Unnormalized Distributions via Noise-Contrastive Estimation
By Geršgorin's 's theorem (Horn & Johnson, 2012, Theorem 6.1.1), the largest eigenvalue of matrix is upper bounded by the largest absolute row sum column sum. Therefore, have
Xmax (2LL Ence (0)) S max
Σ 00,0,27 (0)
)
max Σ
g (es t wiw.y (Po)] + Egg le (eeee (p)]
S max t maxx
sup Ince, (,aee, (po) + sup 212) x,0
x, Snce,f (n(()
(2) B
S pax
f.ce/.d
_(2) ).c...n)
V
We note that Shah et al. (2021b, Lemma 3.1) shows that the projected gradient descent algorithm returns an e-optimal solution for GlobalGISO in polynomial optimization complexity, based on the similarly established smoothness of GlobalGISO, We can establish similar optimization complexity guarantee, but we omit the statement.
G. Experiments
In this section, we present a preliminary empirical evaluation a selected set of estimators on a synthetic data, following a setting in (Shah et al., 2023, Section 5.1). We consider a unnormalized exponential family model
Φc(x) _ (xp(x
where 8 E RPXP for x E 1P. The data generating distribution is chosen as the model with @ = 8" defined as
0 - vP if = 1, or 1, or = j
otherwise.
The samples were generated by brute-force sampling by discretizing each axis by 100 bins. We generated N 10 samples for p E {11, 13, 15, 17, 19} and computed the estimates for each estimator with varying sample size (0.04N, 0.08N,..., 0.64N.. We repeated the experiments with random subsamples for 5 times for each configuration.
Assuming the parameter space θ is bounded under the Frobenius norm, we consider NCE estimators regularized by the Frobenius norm and optimized via gradient descent. We used a regularization weight A, = 10 and a learning rate 0.1 across all settings, except for the flog-NCE estimator, where we used 1) 1.0. Each optimization was run for 1000 gradient steps. As shown in Figure 2, the selected estimators exhibit an empirical convergence rate of n 1122 However, we observed that the fi -NCE estimator (asymmetric log NCE; see Table 1) and the CNCE estimator did not display convergent behavior, despite the theoretical guarantees available for this example. This discrepancy highlights the need for further investigation into the empirical behavior of various estimators, particularly in nigh-dimmniional settings.
LagNCE
Asymmatiic Power (c) NCE (ar - 0.20)
Asymmetric invere log NCE
a-GertNCE (a 6.5)
16-
il,
11,
11,
13, a
12, i
10
10
10"
10a
10
Figure 2. Convergence rate of different NCE estimators.
31


Reviews:
[{"review_id": "VVD2FOgA4h", "reviewer": "Reviewer_7ihf", "summary": "The paper provides a unified perspective on noise-contrastive estimation (NCE) methods for learning unnormalized distributions, integrating several previously separate approaches under a common framework. It introduces two new variants: $\\alpha$-centered NCE ($\\alpha$-CentNCE) and $f$-conditional NCE ($f$-CondNCE), which generalize prior estimators and clarify their relationships. The analysis also revisits the theoretical properties of $f$-CondNCE, challenging prior claims about its connection to score matching.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s effort in reviewing our manuscript. We will incorporate the constructive feedback in our revision. \n\n### Comments on Weaknesses\n\nWe acknowledge that our work primarily focuses on the theoretical unification of different estimators. As suggested by the reviewer, however, experiments involving certain exponential family distributions would indeed provide valuable insights into the behavior of the proposed estimators under various choices of $f$ and/or $\\\\alpha$, as well as for unbounded distributions that go beyond the boundedness assumption in our analysis. In our revision, we will include experiments in the Appendix on the convergence rates of the estimators for both bounded and unbounded distributions, as demonstrated in (Shah et al., 2023). Although a thorough study of the trade-offs associated with different choices of $\\\\alpha$ or $f$ warrants a separate investigation, we will include these experiments to offer an initial indication of such behaviors.\n\n### On Suggestions\n\nWe thank the reviewer for the thoughtful suggestions following a careful reading of our manuscript. We will incorporate these points in our revision, including proof sketches in the main text using an additional page.", "claims_and_evidence": "Theorems and propositions accompany the claims. I read through the remarks and comparisons but did not check the detailed proofs.", "methods_and_evaluation_criteria": "Formulating the problem as an instance of Bregman divergence minimization makes sense, as it appears to align with the existing approaches in the literature.", "theoretical_claims": "The main paper does not appear to contain proof sketches or outlines. Detailed proofs in the appendix should essentially follow the work of Shah et al. (2021b, 2023) and are left unchecked.", "experimental_designs_or_analyses": "The paper does not contain experimental/numerical evaluations.", "relation_to_broader_scientific_literature": "The paper builds on a rich body of noise-contrastive estimation (NCE) work. It generalizes and unifies several estimators previously developed in the research communities. For example, it connects MLE (Fisher, 1922), MC-MLE (Geyer, 1994), and GlobalGISO (Shah et al., 2023) as \"special cases\" of its proposed $\\alpha$-centered NCE framework. Finally, the paper provides a finite-sample convergence guarantee for these estimators.", "essential_references_not_discussed": "I'm not an expert on this specific line of research, but the current literature discussion looks good to me.", "other_strengths_and_weaknesses": "Here are the strengths:\n- The paper presents a unified theoretical framework for NCE-based estimators, connecting prior methods such as MLE, MC-MLE, GlobalGISO, and pseudo-likelihood estimation within a single perspective (and objective).\n- The work corrects a misconception about CondNCE and score matching, showing that its variance diverges in the small-noise regime.\n- The paper establishes finite-sample convergence guarantees, while most prior works provided only asymptotic consistency results.\n\nThere are also some weaknesses:\n- The paper is highly theoretical and does not have empirical validation. Since we have a concrete (convex) optimization objective here, I wonder if the authors could find an exponential family and conduct experiments for, say, different $\\alpha$ values.\n- The finite-sample convergence rates rely on bounded statistics assumptions, which may limit applicability to real-world unnormalized models. The authors also pointed this out in the final discussions.\n- The claim that $\\alpha$-CentNCE unifies multiple estimators lacks a discussion of practical trade-offs. For example, under what conditions would one prefer the estimator in the middle of the second row of Table 2 over others?", "other_comments_or_suggestions": "- Formulas and tables often do not appear on the same page where they are referenced. Maybe this is a property of the paper template, but having them on the same page would be nice.\n- Adding proof sketches to help readers understand the underlying connections among the theorems/claims would be beneficial.\n- On page 1, right panel, line 13 reads, \"The motivation ... is (is) to ...\"", "questions_for_authors": "Please look and respond to the questions and suggestions in the earlier sections. Thanks!", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C1NI5R7kwE", "reviewer": "Reviewer_H2zi", "summary": "This paper presents a unified framework for learning unnormalized distributions through noise-contrastive estimation (NCE), introducing two variants: alpha-CentNCE and f-CondNCE. It demonstrates that alpha-CentNCE generalizes existing methods like MLE, MC-MLE, and GlobalGISO, while f-CondNCE reveals limitations in prior connections to score matching, showing diverging variance in small-noise regimes. The analysis provides novel finite-sample convergence guarantees for exponential families, establishing theoretical foundations for several NCE-based estimators previously studied in isolated contexts.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their effort in reviewing our manuscript and for the valuable feedback. Below, we provide responses to the questions.\n\n* We highlight that the discussion in Section 3 does not assume exponential family distributions, except in the comparison with GlobalGISO, which is specifically designed for exponential family distributions. (We remark that GlobalGISO requires the underlying model to be an exponential family distribution in order to assume that a certain statistic is analytically computable.) However, as pointed out by the reviewer, our current asymptotic and finite-sample analysis is only applicable to bounded exponential family distributions. In Section 5, we discuss the technical challenges of extending the results to unbounded distributions under `Beyond Bounded Exponential Families`. Analyzing this type of estimator for energy-based models beyond exponential family distributions is an important but open problem.\n\n* While our Theorem 3.4 reveals the asymptotic behavior of the empirical conditional NCE objective, we currently do not provide practical guidance on the choice of the noise distribution. As we mention in Remark 3.4, the statistical noise in the empirical conditional NCE objective can potentially be controlled by using a large number of *slicing vectors* (which essentially corresponds to samples from the conditional distribution). We believe this direction merits a separate investigation and leave it for future work.", "claims_and_evidence": "The claims are supported by explicit connections to established methods like MLE and GlobalGISO via Theorem 3.1. The analysis leverages rigorous optimization principles (Bregman divergences) and builds convergence guarantees for exponential families using prior frameworks (Shah et al., 2023), offering a structured unification of disparate estimators through NCE variants.", "methods_and_evaluation_criteria": "The proposed methods are well-aligned with the problem of learning unnormalized distributions. By showing that alpha-CentNCE recovers MLE (Fisher, 1922), MC-MLE (Geyer, 1994), and GlobalGISO (Shah et al., 2023) as special cases (Theorem 3.1), the authors demonstrate consistency with established estimators for exponential families. Similarly, connecting f-CondNCE to pseudo-likelihood (Besag, 1975) and ISO (Vuffray et al., 2016) aligns with prior theoretical frameworks for Markov random fields. The use of Bregman divergences further roots the approach in foundational optimization principles, ensuring compatibility with existing density ratio estimation literature (Sugiyama et al., 2012). These connections validate the methods’ coherence with both classical and modern paradigms.", "theoretical_claims": "I’ve taken a brief look at the proof,and it seems fine for now.", "experimental_designs_or_analyses": "This is a theoretical paper, so no experiments need to be reviewed.", "relation_to_broader_scientific_literature": "The paper proposes a unified NCE-based framework that connects disparate estimators (e.g., MLE, pseudo likelihood, GlobalGISO) as special cases of alpha-CentNCE or localized NCE, resolving their fragmented theoretical treatment. This view aligns with and generalizes prior results, such as Shah et al. (2023)’s GlobalGISO analysis and Besag (1975)’s pseudo likelihood, while clarifying their implicit ties to contrastive learning. Building on this framework, the authors derive new results, including finite-sample convergence rates for NCE estimators and a counterexample showing f-CondNCE’s variance diverges in small-noise regimes—contradicting earlier claims of equivalence to score matching. The synthesis enables systematic extensions of classical methods under a single theoretical lens.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "This paper is not well-written and is somewhat difficult to follow.", "other_comments_or_suggestions": "typo:\n-  On page 1, in the right column, line 4: there is a double \"is.\"", "questions_for_authors": "Your unified framework elegantly connects existing estimators (e.g., MLE, pseudo-likelihood, ISO) via alpha-CentNCE, but how generalizable is this unification to non-exponential family models? For instance, do the finite-sample convergence guarantees for alpha-CentNCE extend to broader classes of unnormalized distributions, such as energy-based models with deep parameterizations, or are they inherently limited to bounded exponential families? \n\nAdditionally, while your analysis reveals divergence in f-CondNCE’s variance under vanishing noise, what practical guidance does this imply for choosing noise distributions in real-world applications?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lES9o3X5NF", "reviewer": "Reviewer_46T4", "summary": "This paper provides a unified perspective on various estimators for learning unnormalized distributions (also known as energy-based models) using Noise-Contrastive Estimation (NCE). Specifically, they introduces $\\alpha$-Centered NCE ($\\alpha$-CentNCE) and f-Conditional NCE (f-CondNCE) as generalized versions of traditional NCE.\n\nBuilding on this, their contributions include\n1. they show that estimators such as Maximum Likelihood Estimation (MLE), Monte Carlo MLE (MC-MLE), and GlobalGISO can be viewed as special cases of $\\alpha$-CentNCE.\n2. f-Conditional NCE (f-CondNCE) generalizes previous conditional noise-contrastive estimation methods and clarifies their relationship with score matching.\n3. they obtain finite-sample convergence rates for different estimators under regularity assumptions, which are novel for most NCE-based estimators.\n4. they provide conditions under which these estimators achieve the parametric rate of convergence.\n\nThe paper is mainly a theoretical paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s careful assessment of our manuscript and their constructive feedback. Below, we respond to the points raised under *Weaknesses* to clarify our contributions. In summary, we believe that all the issues raised are important and warrant separate, dedicated future investigations.\n\n* As pointed out by the reviewer, studying the $\\alpha$-CentNCE and its advantage over the NCE counterpart $f_\\alpha$-NCE is a valuable direction for future work. We leave this for subsequent study.\n\n* As noted in Remark 3.3 and Remark 4.3, $f$-CondNCE suffers from variance explosion. However, our finite-sample analysis assumes that a single conditional sample $y_i$ is drawn for each data point $x_i$ (see line 201). As alluded to in Remark 3.4, this variance may be controlled by using a large number of conditional samples to reduce statistical noise. This reflects a fundamental trade-off between computational efficiency and statistical accuracy. A more thorough analysis of this trade-off is left for future work.\n\n* Choosing an optimal noise distribution for NCE is an important but open problem in the literature; see, for example, the recent study by Chehab et al. (UAI 2022). We will add a paragraph discussing this point to clarify the current understanding in the literature.\n\n* As the reviewer noted, we assume bounded exponential family distributions throughout. This assumption is fairly standard in the theoretical analysis of such estimators to enable tractable analysis; see, for example, (Shah et al., 2023). One possible justification is that many real-world distributions have bounded statistics and can thus be modeled by bounded exponential family distributions. That said, extending the analysis to unbounded distributions is of theoretical interest and also important for handling heavy-tailed data. In Section 5, we remark on the technical challenges involved in such an extension under `Beyond Bounded Exponential Families`. We leave this theoretical direction for future work.\n\n* Lastly, as the reviewer pointed out, for NCE-type arguments to work well in practice, using a data-adaptive noise distribution is crucial, as a mismatch between the noise and data distributions can significantly degrade performance. A theoretical investigation into data-adaptive choices of noise distributions is another compelling direction for future research.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "This paper is not a methodology focusing paper but rather a theoretical paper. Their proposed framework of $\\alpha$-Centered NCE ($\\alpha$-CentNCE) and f-Conditional NCE (f-CondNCE) are sound with supporting theoretical results.", "theoretical_claims": "I haven't been able to check every detail of the proof but the proofs of their main theorems 4.1-4.3 are correct to me.", "experimental_designs_or_analyses": "The paper has no experiments. It is a pure theoretical paper.", "relation_to_broader_scientific_literature": "The problem of estimating unnormalized probability distributions that this paper focuses on is fundamental in energy-based models (EBMs) and appears in various fields, including generative modeling, density estimation, and graphical models. There exist many standard estimators including Maximum Likelihood Estimation (MLE), Monte Carlo MLE (MC-MLE), and GlobalGISO. For them, the paper reframes these seemingly different estimators under a unified NCE-based framework, showing that various inference methods can be viewed as special cases of α-Centered NCE (α-CentNCE). Moreover, the paper also connects to the score matching (SM), introduced by Hyvärinen (2005), provides an alternative estimation principle that avoids the computation of partition functions. Previous research (Ceylan & Gutmann, 2018) suggested that f-CondNCE approximates Score Matching in the limit as noise approaches zero while this paper challenges this claim, showing that f-CondNCE does NOT converge to Score Matching but instead exhibits diverging variance in the small-noise regime.", "essential_references_not_discussed": "I am not an expert in this area but to my best knowledge, the authors tried their best to relate works that are essential to understanding their key contributions.", "other_strengths_and_weaknesses": "Strengths:\n1. They paper theoretically unifies learning methods for unnormalized models including different estimation techniques (MLE, MC-MLE, Score Matching, etc.) while these estimators have only been developed independently, often with limited understanding of their connections until this paper.\n2. It also introduces f-Conditional NCE (f-CondNCE), which extends conditional NCE and clarifies its theoretical behavior. Importantly, f-CondNCE corrects a misleading interpretation in prior work (Ceylan & Gutmann, 2018), showing that it does NOT converge to Score Matching in small-noise regimes.\n3. The paper establishes finite-sample convergence rates for a broad class of NCE-based estimators, including $\\alpha$-CentNCE and f-CondNCE. Moreover, they showed that most of these estimators achieve a parametric convergence rate of O(n^{-1/2}), which is the best possible rate in standard statistical settings.\n\nWeakness:\n1. The paper suggests that $\\alpha$-CentNCE offers advantages over traditional Noise-Contrastive Estimation (NCE) in certain cases, but it does not fully formalize when and why $\\alpha$-CentNCE is strictly better (while I only believe this is more a future direction rather than being completely solved in this one paper). For example, the paper proposes that centering may reduce variance but does not formally prove how variance changes as a function of $\\alpha$. Studying this would provide guidance on how to pick $\\alpha$ in practice. I would assume it involves some bias-variance tradeoff analysis.\n2. The paper proves that f-CondNCE suffers from variance explosion in the small-noise limit, debunking prior claims, while it leaves an open theoretical problem: Can a modified version of f-CondNCE control variance growth while still approximating Score Matching?\n3. The paper mentions that noise selection is crucial for the performance of NCE-based methods. However, the paper does not derive any results on optimal noise selection or analyze how different choices of $q_n(x)$ affect statistical efficiency.\n4. I am not sure if these are standard assumptions in this area but The paper assumes that the sufficient statistic $\\psi(x)$ is bounded while in commonly used models like Gaussian models $\\psi(x)=x$ and log-linear models where $\\psi(x)=\\log(x)$, they are not. Also, the paper assumes that the noise distribution $q_n(x)$ is fixed and independent of the learned model while I believe in practice, this noise distribution should be tuned and selected dependent on the data.", "other_comments_or_suggestions": "NA", "questions_for_authors": "Please see the four questions listed in \"Other Strengths And Weaknesses\".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "NA"}}, {"review_id": "8ofsed8k13", "reviewer": "Reviewer_SNf8", "summary": "In this work the authors present a unification of noise contrastive estimation (NCE) losses. In particular they consider a class of risks based on optimizing the density ratio of the model density compared to a known noise density (canonically a uniform distribution on a set known to contain the support of the true distribution) with a convex function in a Bregman-Divergence-style loss. From here they consider \"$\\alpha$-centered NCE\", where the risk is normalized in some sense, and \"$f$-conditional NCE\" where the noise density is based on the training samples. \n\nFor these settings they prove that the population version of these losses will indeed work in a realizable modeling scenario (Prop 1.1, Prop 2.1, Prop 2.2). I addition they show that flexible forms of these losses using the correct loss parameters includes several well-known NCE losses, showing that the framework presented covers useful cases (Theorem 3.1, 3.2) and follow up with asymptotic and finite sample analysis (middle of section 3 and onwards).\n\n## Update after rebuttal:\n\nNothing to update, I keep my score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their effort in reviewing our paper. We will incorporate all the comments from the reviews and revise our manuscript accordingly.", "claims_and_evidence": "This is a purely theory paper so this isn't so applicable.", "methods_and_evaluation_criteria": "N/A", "theoretical_claims": "I have not checked the proofs carefully, but scanned them, and nothing immediately stands out as being problematic.", "experimental_designs_or_analyses": "N/A", "relation_to_broader_scientific_literature": "This does indeed seem to be a nice framework covering many NCE estimators with have previously been explored and includes some nice general results on the behavior of these. As stated in the intro, this covers MLE, MC0MLE, Global GISO, along with others.", "essential_references_not_discussed": "None that I am aware of.", "other_strengths_and_weaknesses": "Overall the paper seems very strong to me. It is a natural sort of paper which generalizes many previous results and expands upon them.", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a theoretical framework that unifies existing methods such as MLE, MC-MLE, and GlobalGISO for learning unnormalized distributions via noise-contrastive estimation (NCE). The reviewers appreciated the conceptual clarity, the finite-sample guarantees, and the correction of misconceptions in prior work. While the paper is theoretical in nature and does not include empirical results, we find the contributions to be meaningful and significant. We therefore recommend acceptance. In the revision, please supplement the paper with experimental results and outlines of the main proofs.
Decision: Accept (poster)