``Noisier'’ Noise Contrastive Estimation is (Almost) Maximum Likelihood
Authors: Peiyu Yu
Dinghuai Zhang
Hengzhi He
Xiaojian Ma
Sirui Xie
RUIYAO MIAO
Yifan Lu
Yasi Zhang
Deqian Kong
Ruiqi Gao
Jianwen Xie
Guang Cheng
Ying Nian Wu
Venue: ICLR 2026 Poster (2026)
Keywords: Noise Contrastive Estimation; Generative Models
Primary area: generative models
TL;DR: We revisit NCE through the lens of approximate maximum likelihood, showing that a noisier NCE better approximates MLE and provides strong methods for image modeling including distillation, anomaly detection, and offline black-box optimization.

Abstract: Noise Contrastive Estimation (NCE) has fueled major breakthroughs in representation learning and generative modeling. Yet a long-standing challenge remains: accurately estimating ratios between distributions that differ substantially, which significantly limits the applicability of NCE on modern high-dimensional and multimodal datasets. We revisit this problem from a less explored perspective: the magnitude of the noise distribution. Specifically, we show that with a virtually scaled (i.e., artificially increased) noise magnitude, the gradient of the NCE objective can closely align with that of Maximum Likelihood, enabling a trajectory-wise approximation from NCE to MLE, and faster convergence both theoretically and empirically. Building on this insight, we introduce "Noisier" NCE, a simple drop-in modification to vanilla NCE that incurs little to no extra computational cost, while effectively handling density-ratio estimation in challenging regimes where traditional MLE and NCE struggle. Beyond improving classical density-ratio learning, "Noisier" NCE proves broadly applicable: it achieves strong results across image modeling, anomaly detection, and offline black-box optimization. On CIFAR-10 and ImageNet64×64 datasets, it yields 10-step and even 1-step samplers that match or surpass state-of-the-art methods, while cutting training iterations by up to half.

Published as a conference paper at ICLR 2026
"NoISIER" NOISE CONTRASTIVE ESTIMATION IS (AL- MOST) MAXIMUM LIKELIHOOD
Peiyu Yu!-
Dinghuai Zhang?*
Hengzhi He
Xiaojian Ma*
Sirui Xie
Ruiyao Miao!
Yifan Lu
Yasi Zhang'
Deqian Kong'
Ruiqi Gao
Jianwen Xie
Guang Cheng
Ying Nian Wu
University of California, Los Angeles
Mila Quebec Al Institute GGoogle DeepMind
Beijing Institute for General Artificial Intelligence (BIGAT)
AAkool Research
ABSTRACT
Noise Contrastive Estimation (NCE) has fueled major breakthroughs in represen- tation learning and generative modeling. Yet a Long-standing challenge remains: accurately estimating ratios between distributions that differ substantially, which significantly limits the applicability of NCE on modern high-dimensional and multimodal datasets. We revisit this problem from less explored perspective: the magnitude of the noise distribution. Specifically, we show that with a virtually scaled (i.e., artificially increased) noise magnitude, the gradient of the NCE objec- tive can closely align with that of Maximum Likelihood, enabling trajectory- wise approximation from NCE MLE, and faster convergence both theoretically and empirically. Building on this insight, we introduce "Noisier" NCE, a simple drop-in modification to vanilla NCE that incurs little to no extra computational cost, while effectively handling density-ratio estimation in challenging regimes where tradi- tional MLE and NCE struggle. Beyond improving classical density-ratio learning, "Noisier" NCE proves broadly applicable: it achieves strong results across image modeling, anomaly detection, and offline black-box optimization. On CIFAR-10 and ImageNet64x64 datasets, it yields 10-step and even 1-step samplers that match or surpass state-of-the-art methods, while cutting training iterations by up to half.
INTRODUCTION
Generative modeling of data distributions has made impressive progress in recent years, driven by the development of a rich family of deep generative models (Ho et al., 2020; Song et al., 2020; Rombach et al., 2022; Karras et al., 2022: Lipman et al., 2022; Liu et al., 2022). Among existing paradigms, Noise Contrastive Estimation (NCE) (Gutmann & Hyvärinen, 2010; 2012), also referred to as reato estimation some literature (Sugiyama et al., 2012), is powerful and fundamental framework that unifies generative modeling with discriminative representation learning (Karras et al., 2019; Chen et al., 2020; Radford et al., 2021; Huang et al., 2025). NCE reduces density estimation to classification task: learning the ratio r(x) q. (x)//((()) between target distribution q. and noise distribution from samples of each, thereby enabling density estimation without explicitly modeling the target density itself (Gutmann & Hyvärinen, 2010; Sugiyama et al., 2012).
Despite its wide adoption, most existing NCEE-based objectives suffer from a fundamental drawback: when the target and noise distributions differ substantially. the neural classifier can achieve near- perfect discrimination accuracy while still providing poor estimate of the density ratio. This issue, often referred as the density-chasm (Rhodes et al., 2020), arises when the gap between the two distributions is large, e.g., when the KL divergence between q. and go exceeds tens of nats. Such situations are common in modern high-dimensional highly multimodal datasets. Although NCE
"qqual Contribution. Code and data available at http://ghhhh.cmmyyy/PPyyo////eee

---

Published as a conference paper at ICLR 2026
Absolute gradlent etror noise magritude
(ine Nerd
MC2 (4-30
menn
35
VYT alwd)
1300
0.35
23
0.00
0.05
00
420 Noise magnitude
1002
Peration
(b)
Figure 1: "Noisier" NCE gradients approach MLE gradients. As a sanity check, we simulate results using 2d Gaussian distributions; true MLE gradients can analytically computed. In Fig. 1a, we can see that M + oo Icads to trajectory-wise convergence from NCE to MLE. In Fig. lb, as the noise magnitude M increases, "noisier" NCE gradients V. LM CNCE, approach MLE gradients VoJ P.JTMER bias decaying in the order of O which is consistent with Proposition 3.3. Further details can be found in Appx. D.I.
estimators are asymptotically consistent, meaning they recover the true density ratio in the infinite- sample limit, the convergence rate is provably slow: even an exponential increase in sample size yields only a linear decrease in estimation error (Poole al., 2019; McAllester & Stratos, 2020), and the issue persists even with infinite data (Liu et al., 2021).
In this paper, revisit NCE from less explored perspective: the magnitude of the noise distribution used for ratio estimation. We show that, under mild conditions, introducing a virtually scaled noise magnitude allows the gradient of the NCE objective align with that of Maximum Likelihood Estimation (MLE). This observation has two key implications. First, establishes a principled gradient-level connection between NCE and MLE, where the NCE gradient serves controlled approximation to the MLE gradient. This perspective highlights that NCE can be understood as approximating the MLE in terms of the optimization trajectory rather than merely matching asymptotic етror rates established by Gutmann & Hyvärinen (2012). Second, it offers new lens on the density-chasm problem, revealing that "noisier" NCE naturally mitigates convergence issues. Building on this insight, we propose "Noisier" Noise Contrastive Estimation (N'CE), simple drop- modification to vanilla NCE that requires little to no additional computational cost. we show in Sec. 4, this framework proves broadly applicable, demonstrating strong empirical performance in various downstream tasks (see Fig. for an illustrative example).
Our main contributions are as follows: i) Theoretically, show that, under mild conditions, "noisier" NCE objective yields gradients that align with those of maximum likelihood, thus bridging the gap between NCE and MLE at the optimization trajectory level. Further, analysis reveals that a virtually scaled noise magnitude naturally alleviates the convergence issues that arise when the target and noise distributions differ substantially. ii) Practically, we propose NºCF, drop-in modification to vanilla NCE with negligible computational overhead. NºCE achieves strong empirical performance across diverse tasks including image modeling, anomaly detection, and offline black-box optimization. It produces 10-step and even 1-step samplers that match or surpass state-of-the- methods on the CIFAR-10 10 and ImageNet64x66 datasets, with up to half the training cost.
BACKGROUND
(NWJ) Noise Con- trastive Estimation (NCE) (Gutmann & Hyvärinen, 2010; 2012) offers discriminative route to training unnormalized statistical models, sidestepping the need for direct likelihood maximization. Let q+ denote the target distribution and qo known noise distribution. Rather than directly modeling Pa to approximate qu, NCE estimates the ratio ra (x) between samples from {s and go- When the model correctly specified, the optimal ratio satis- fies fo- g./go. which in turn implies Pa- z q.. In practice, Ta is often parameterized as the exponential of a scalar-output neural network fa. This parameterization naturally aligns with the energy-based view of Pa (LeCun al., 2006; Xie al., 2016; Du & Mordatch, 2019), where fa
Noise Contrastive Estimation (NCE) and Sguyen-Wainnright
(x(x)
4n(x) by training classifier to distinguish
2

---

Published as a conference paper at ICLR 2026
incorporates the partition function. A common choice of classification loss is the logistic loss:
Ta (xx
((x) Ex
log
Ex
log + [x]
r+ (x)
Numerous extensions of NCE have been developed, including generalized loss functions (Pihlaja al., 2010; Sugiyama et al., 2012: Menon & Ong, 2016; Poole et al., 2019; Liu et al., 2021) and multi-stage ratio estimation strategies (Rhodes et al., 2020; Xiao & Han, 2022), which greatly enrich the paradigm. In this work, we revisit NCE from the perspective of the noise distribution's 's magnitude, showing that virtually scaling the noise magnitude offers a simple, effective remedy to long-standing convergence issues with little to no computational overhead.
Being equally fundamental and seminal, Nguyen-Wainwrige Wainwright-lordan (NWJ), addresses the ra- tio-estimation problem through a variational characterization of f-divergences (Nguyen et al., 2010). This variational view reveals that likelihood-ratii estimation can be performed by maximizing convex risk functional, yielding M-estimators that are statistically consistent and minimax-ov opptimal under suitable regularity assumptions. Algorithmically. NWJ induces a tractable objective of the form
(2) where To = log fo. Provably, the optimum critic obtained by maximizing Eq. (2) retrieves the optimal ratio estimator. In contrast to NCE, which originates from a binary classification surrogate, NWJ is derived directly from convex duality and makes no explicit reference to a discriminative viewpoint. Intriguingly, however, as we demonstrate in Sec. 3.4, the noise-scaled NCE objectives studied in this work establish implicit connection between these ostensibly distinct paradigms. This perspective positions our approach as unifying family of likelihood-ratio estimators.
NWW (a) = Ex-q. [Ta (x)] + Ex-qu [exp (exp (x)],
Maximum Likelihood Estimation (MLE) Maximum Likelihood Estimation (MLE) remains the generative route: it fits probabilistic models by maximizing E.g [log Pa (x)] over data samples. A common parameterization of Pa (LeCun et al., 2006; Xie et al., 2016: Du & Mordatch, 2019)
Pa (x)
exp (Sa (x)) go(x); Za
where fax denotes the unnormalized log-density, Za the partition function, and (u(x) is the base noise distribution. The model in Eq. (3) can be interpreted as an enare correction or exponential tilting of qo (Xie et al., 2016). The corresponding gradient takes the general form
V (a) E., (Vafa - Ep. V (x)].
(4)
The first expectation over the data distribution is straightforward requires sampling from Pa- This is often intractable when the model involves an unknown partition function. In such cases, Markov Chain Monte Carlo (MCMC) methods such as Langevin dynam- ies (Welling & Teh, 2011) are commonly employed. While theoretically valid, MCMC sampling can converge slowly in high-ddmennsionl or multimodal settings, making MLE difficult to apply in practice (Nijkamp et al., 2020). This sampling bottleneck is a key motivation for alternatives such as NCE, which avoids direct sampling from Pax by reframing density estimation as a classification problem idea we revisit from a less- exs-ooxd perspective in the next section.
approximate, but the second
3
"NoISIER" NOISE CONTRASTIVE ESTIMATION
3.1 NOISE MAGNITUDE
We now make precise what we mean by the magnitude of the noise distribution in CºCE. Recall the standard objective in Eq. (1), where samples from the base distribution qo serve as negatives. More generally, we may scale the contribution of go by positive factor M > 1, which we refer to as the noise magnitude. This leads to the following "noisier" NCE objective (Gutmann & Hyvärinen, 2012):
(x)
M
(a((a) Eg)()))
log
log M Mrrr(*
(5)
M (x)(xx)
Here, M = recovers the standard NCE objective, while larger corresponds to amplifying the effective weight of the noise distribution. Intuitively, this adjustment can be seen replacing qo with a virtually scaled mixture of M independent copies of qu- As we show below, increasing M has a striking effect: under mild conditions, the gradient of Las(cu (ax approaches that of MLE, ie., Eq. (4). This not only establishes a principled bridge between NCE and MLE, but also provides a simple and effective remedy for the convergence issues of standard NCE.
3

---

Published as a conference paper at ICLR 2026
3.2 SCALING EFFECT OF NOISE MAGNITUDE
Proposition 3.1 (Gradient approximation). Under mild regularity conditions.
lim Va (m)(a) = Ey. (a.laaaaat Epa (x)]
M-C
Sketch of Proof. The gradient of Las be expressed
M / M (x) (1. -(xx)-aax)) V (xla(x)
M
Va (ax(())
This makes clear that, with smooth densities and ratio functions, as M grows, the weight converges 1 and the gradient approaches the MLE form. Additional derivation details appear in the Appx. B.1.
Remark. Our result (Proposition 3.1) establishes gradient-level approximation MLE. Prior works have not analyzed this: Gutmann & Hyvärinen (2012) focused only on asymptotic consistency without optimization dynamics, while Mnih & Teh (2012) treated discrete embeddings without convergence analysis. In contrast, our formulation applies to continuous settings and, crucially, enables the first explicit convergence guarantees for "noisier" NCE-type objectives. We demonstrate this in the exponential-fame faamily case summarized in Theorem 3.2.
Theorem 3.2 (Convergence in exponential families (informal)). Under standard regularity
assumptions for exponential faminii (details in Appx. B.3), letting Amin Amax denote the extremal eigenvalues of the Fisher information matrix, then for sufficiently large M, normalized gradient ascent on Eq. (5) finds an iterate within distance å of the true parameter of in at most
Amnax
1|a0 ax xl
T PSC C
Amia
62
iterations for universal constant C i.e. there exists & S T with |o 1|2
Remark. Theorem 3.2 gives a polynomial iteration complexity in the condition number Amax/Ami /min for exponential families when M sufficiently large. In contrast, for standard NCE the effective Hessian can be ill-conditioned unless 9. and qu already close. This can lead to much worse, often effectively exponential, dependence on the gap. Our result shows that virtually scaling the noise magnitude M acts form of landscape regularization: the Hessian condition number of Eq. (5) remains uniformly bounded under standard assumptions (Liu et al., 2021), withou requiring distributional closeness. We validate this intuition empirically: in 2D Gaussian simulations (Fig. the trajectories indeed converge as predicted, and similar behavior persists even in high- nigh-dimensional, multimodal neural settings (Sec. 4). full formal statement and proof are given in Appx. B.3.
3.3 PRACTICAL ERROR ANALYSIS, NCCE FAMILY AND REGULARIZATION
The population results above assume exact expectations and arbitrarily large M. In practice both M and the sample sizes finite. We therefore ask: how well does the empirical "noisier" NCE gradient approximate the MLE gradient under these practical constraints? Proposition 3.3 provides an error decomposition that makes this trade-off explicit:
Proposition 3.3 (Finite-M finite-sample error (informal)). Let L.M be the empirical objective
buili from n i.i.d. samples from q. and qo, and write DM(x) (x) Under standard regularity assumptions (see Appx. B. B,2), the mean mean-squan squared approximation error satisfies
rx(x)
EE V J MLE (a) vaIma()) s V. Bu, where
C B, 0 1 Vu vu= M2
( lograll + minf M² E Va log 12, Enpl VaTa 12)
n
and C hides benign constants.
4

---

Published as a conference paper at ICLR 2026
Remark. By is the finite-M bias; Vu is the sampling variance. The finite-M bias decays as O(1/M²), while the variance can grow as OMM2/n) unless the ratio (or its log) is sufficiently smooth under do, in which case the variance term saturates the y Vr ll level. This highlights bias-variance trade-off' shaped jointly by M and the behavior of To. Importantly, it shows that NºCE naturally induces a continuum of empirical objectives parameterized by M, and that an optimal finite- M estimator exists in this spectrum. Besides, controlling the roughness of the ratio can further help stabilize the variance term and obtain robust performance. We make the optimal-M characterization precise below, and present two concrete regularizations that restrict To accordingly. As shown in Sec. 4, coupling these practical choices with the NCCE objective yields strong performance across a diverse set of tasks. A formal statement for Proposition 3.3 with more details appears in Appx. B.2.
The C2 family and U-shaped patterns Indeed, with a finite fixed sample size n, we consistently observe the U-shaped dependence on M predicted by Proposition 3.3. This emerges clearly in controlled 5-dimensional Gaussian experiments across different regimes (Appx. D.I), and perhaps more impressively, reappears in digh-dimessional neural settings (Tabs. 16, 17 and 26). Furthermore, Proposition 3.3 predicts that the optimal M should scale no larger than Cyn, for some C' expected to lie within - 10 determined by the actual behavior ra. This theoretical prediction matches our empirical findings with remarkable fidelity. Thus, the finite-sample analysis in Proposition 3.3 both explains and anticipates the observed U-shaped curves, offering a principled and practical guideline for selecting M in real-world applications. We next introduce two regularization strategies that provide practical handles for controlling the behavior of To.
Multi-stage ratio estimation One way to stabilize the gradient approximation is to adopt the multi- stage estimation strategy of Rhodes et al. (2020). This is especially useful when (i) a convenient base noise distribution is available, (ii) prefer use the density model directly decision making rather than as a smooth reward, and (iii) the problem scale is low- or moderately high-dimensional. The key idea is to decompose the ratio between q. and go into telescoping product, 1= 4x 9k
discriminator for
21 where {4 are pre-specified intermediate distributions. Each ratio now involves pair (m=-1, 9k) with greater overlap by design, yielding smaller values and more controllable variance at each stage. The trade-off is computational: in high-dimensional settings many stages may be needed, increasing overhead. For this reason, we mainly apply this technique in lower-dimensional tasks such as latent-space modeling.
şo gic gK--1 go
Direct ratio regularization A more general and convenient strategy is add penalty on the ratio itself, for example E|| logra 113. directly to the "noisier" NCE objective. This approach is broadly applicable to digeessiinni data and does not rely on auxiliary intermediate distributions. The trade-off is that the added penalty may bias the gradients. Nonetheless, as we show in Sec. 4. this regularization is highly effective in practice, particularly for training reward models or crities in high-dimensional settings such as on ImageNet64 x 64 datasets.
3.4 AN INFORMATION THEORETTI PERSPECTIVE
Complementing the gradient- based and finite-sample analyses, an information-theoreti interpretation further reveals that the NºCE objectives trace continuous path between variational representations of JS and KL divergences (Sec. 7.13 in Polyanskiy & Wu (2025)), thereby making explicit both the M oo limit of NºCE being NWJ and its consistency with maximum likelloood estimation.
Variational representations of f-divergences Specifically, on one end of the spectrum, the KL divergence satisfies the Nguyen-Wainvrihh--orraa (NWJ) representation (see Sec. 2), recovering the NWJ objective in Eq. (2) when T logr: r:
DRr.(a-llgo) = 1 + sup
E. [T()) - Eg |e (xx)) = + Sup (log rx)) -Ego [r()))
(6)
T
On the other end, the Jensen-Shannon divergence can be written as
r ) + Egu log 1 +
Das(g. (go) = log 2 sup LE. log
(7)
which matches the standard NCE objective in Eq. (1) up to additive constant (Nowozin et al., 2016). Thus NWJ and NCE arise as variational lower bounds on KL and JS divergences, respectively, each attaining its optimum at the true ratio y q4/90-
5

---

Published as a conference paper at ICLR 2026
CE objectives trace continuous interpolation path The NºCE objective in Eq. (5) interpo- lates between these two extremes. Let a = M/(1+ + M) E (0, 1) and consider the divergence
Da (g (19o) = (1 DLL + a=)g)) + a Pxu(zoloogo + (1 a)p-),
(8)
which satisfies D1/2- = Das and D. + DKL. as a One can now show that
r M +r +-)) + M Ego
M + )
M
Da(9-1190) = h(a) + sup Eg. log
log
(9)
which is identical (up to h(a), the binary entropy function) to the NºCE objective: (1) at M = 1 Mrr
(a = 1/2), NºCE reduces to the usual NCE/JS bound; (ii) as M + co (a + 1), log M
logr, r, Mog MM + rr, so the Eq. (5) converges to Eg. [logr] g. [r] + const, the NWJ form of M+r
KI.; maximizing this variational bound yields the same population optimum maximum likelihood. We further provide empirical evidence in Appx. D.I that NºCE indeed approaches NWJ when M sufficiently large (M 1e9). With Sec. 3.2, these results jointly explain why letting M co recovers maximum-likelihodd learning, at both the divergence level and the gradient-dyname level.
4 EXPERIMENTS
We organize our empirical study around three main questions: (i) How does the proposed objective compare with standard baselines such as pure MLE and vanilla NCE? (ii) Do the advantages of "noisier" NCE transfer to downstream tasks? (iii) How do key hyperparameters, in particular the noise magnitude M, affect performance?
To answer these, we conduct experiments across a broad range of datasets and benchmarks, span- ning diverse model families. The results consistently show that our method outperforms existing approaches, while providing new insights into the role of noise scaling. Full experimental settings, descripteddooo
implementation details and baseline descriptions are provided in Appx. C and D.
4.1 LEARNING LATENT-SPACE ENERGY- ENERGY-BASED MODEL (LEBM) WITH NºCE
Table 2: AUPRC(T) scores for unsupervised
Table 1: FID(J) on different datasets. We highlight our model, the 1st and 2nd performances; tables henceforth follow this format. Numbers from the first six rows are from Yu et al. (2024). nz denotes the latent dimension.
anomaly detection on MNIST. Baseline numbers are taken from Yoon al. (2023); Yu et al. (2024). Full results with variances found in Appx. D.3.
M and K denote the noise magnitude and num. of stages for ratio estimation, respectively.
Heldout Digit
5
9
AE
0.062
0.204
0,259
0.125
0,113
VAE
0.063
0.337
0.325
0.148
0.104
Model
SVHN
CelebA
CIFARIO
CelebAHO n2-512
ABP
0.095
0.138
0.147
0.138
0.102
a2-100
n2-100
n2-128
IGEBM
0.101
0.106
0,205
0,100
0.079
49.71
$1.50
90.30
160.21
MEG
0.281
0.401
0.402
0,290
0.342
LBEMLEEM"
29.44
37.87
70.15
133.07
BiGAN-7
0.287
0.443
0.514
0.347
0.307
SRI
44.86
61.03
ABP-LEBM
0.336
0.630
0.619
0.463
0.413
SRI (L=5)
35.32
47.95
JVAEBM
0.297
0.723
0.676
0.490
0.383
VAE
42.81
44.40
72.90
Adaptive CE
0.531
0.729
0.742
0.620
0.199
RAE
40.02
40.95
74.16
NAE
0.802
0.648
0.716
0.789
0.441
34.81
47.84
110.37
154.60
MPDR-S
0.764
0.823
0.741
0.857
0.478
MPDR-R
0.844
0.7.1
0.737
0.850
0.569
MLE-LEBM
32.74
40.24
90.54
111.11
DAMC
0.684
0.911
0.939
0.801
0.705
NCE-LEBM
30.71
39.61
92.83
118.84
DAMC-NCE
0.702
0.829
0.764
0.605
0.502
CCCE-EEM
K-100, K-1
26.84
33.05
77.35
101.71
DAMC-N'CE
M-11 10,000 K=3
25.63
31.09
77.05
95.66
M= 100 x=1
0.910
0.911
0.935
0.779
0.699
00-00,22 X-3
0.959
0.935
0.959
0.845
0.854
As a proof of concept, we first conduct a set of lightweight experiments on latent energy-based models (LEBMs) across image datasets including CIFAR-10 (Krizhevsky et al., 2009), MNIST (LeCun, 1998), SVHN (Netzer et al., 201 1), CelebA64 (Liu et al., 2015), and CelebA.aak- (Lee et al., 2020). Learning LEBM (Pang et al., 2020a; Yu et al., 2022) can be viewed fitting an unnormalized density model in the latent space of a VAE (Appx. C.1). Prior approaches largely rely on MCMC-based MLE-style training (Pang al., 2020a; Du et al., 2021), where convergence and

---

Published as a conference paper at ICLR 2026
Table 3: CIEAR-10 (DDPM backbone) and ImageNer64x644 (EDM backbone) results shown side-by-side. First six rows are from Yoon al. (2024). hhighlights the starting point of DxMI fine-tuning.
Backbone: DDPM
Backbone: EDM
Method
NHE
HD (4)
Rec. (1)
Method
NFE
FID (4)
Prec. (7)
Rec. (1)
DDPM
3.21
0.57
(Heun)
79
2.44
FastDPM
35.85
0.29
EDM (Ancestral)*
10
50.27
0.37
0.35
DDIM
10
13.36
Consistency Model
2
4.70
0.69
0.64
SFT-PG
10
4.82
0.606
Consistency Model
I
6.20
0.68
0.63
DAMI
3.19
0.625
D&MI
10
2.68
0.777
0.574
Value Guidance
0.623
DxMI + Guidance
2.67
DxMI + NCE (M-1)
10
3.93
0.623
DxMI + NCE (K-1)
10
2.69
0,756
0.585
DAMI+ + N"CE (0=100)
10
2.99
0.638
D&MI + C"CE (K=100)
10
2.23
0.757
0.599
stability are often problematic. This makes LEBMs a natural and tractable testbed for prototyping our method. In these experiments, we apply ratio decomposition as a regularization (Sec. 3.3) and follow the training and evaluation protocols in Pang et al. (2020a) (see Appx. C.2).
Generative image modeling We evaluate the quality of the learned LEBM with generated images measured by FID scores (Heusel al., 2017) (see Tab. 1). For image generation with LEBM, we perform 100-step short-run Langevin Dynamics (L.D) to draw latent vectors from the learned models, and map the latent vectors to the image space with the decoder network.
We observe that: first, LEBMs trained with Nº (i.e., rows with M=100) consistently outperform those trained with vanilla NCE, with multi-stage estimation providing further gains that validate our analysis. Second, LEBMs trained with NCCE show substantial improvements over those trained with MLE and short-run LD (denoted MLE-LEBM). These results confirm that NCCE is a more reliable and effective objective that addresses the issues inherent to vanilla NCE and MCMC-based MLE. Finally, the improvement becomes more pronounced as the latent dimension nz increases, especially on the CelebAMask-HO dataset with nz=51 2. This suggests that our proposal scales more gracefully to high-dimensional, multimodal targets.
Anomaly Detection To further highlight the practical advantages of our approach in highly mul- timodal settings, we evaluate anomaly detection on MNIST following the setup in Zenati et al. (2018). Specifically, we train models with one digit held out as anomalous, focusing on digits 1, 4, 5, 7, and 9-cases known to be especially challenging. We build on DAMC (Yu et al., 2024) for posterior inference, replacing its original prior with an LEBM learned via N2CE. Since empirical posteriors from DAMC are sharp and highly multimodal, this serves as a demanding testbed. Table 2 reports AUPRC scores averaged over 10 trials. Our method yields consistent and often substantial improvements over baselines, achieving strong overall performance across the hardest digits. Full experimental details appear in Appx. C.2 and D.3.
4.2 REWARD AND CRITIC LEARNING FOR DIFFUSION DISTILLATION
Building on the success of latent-space modeling, we next consider more challenging high- dimensional settings in the ambient image space. In particular, we study two representative scenarios for distilling diffusion samplers: (i) learning energy-based rewards for diffusion fine-tuning (DxMI) (Yoon et al., 2024), and (ii) learning critics for adversarial distillation (SiD?A) (Zhou et al., 2024a). In both cases, we simply replace the original objectives with our "noisier" NCE combined with direct ratio regularization (Sec. 3.3), while using the same architectures and critical hyperparameters as baseline methods. Full implementation details appear Appx. C.3.
In the DxMI setup (Yoon et al., 2024), NºCE yields substantial improvements over vanilla NCE-like variants, previous methods and even the teacher models with full NFE (Tab. 3), underscoring its effectiveness in training rewards for diffusion distillation. In the SID setting (Zhou et al., 2024a), our method greatly outperforms vanilla NCE variants, and not only matches but can surpass strong baselines (Tabs. 4 and 5), notably with up to half the training iterations. Together, these results confirm that NºCE scales to nigh-dimensional tasks while improving training efficiency across distinct distillation regimes. We provide additional ablation studies on M on CIFAR-10 in Appx. D.4.
7

---

Published as a conference paper at ICLR 2026
Table CIFAR-10 results. "FID-U/C (iters)" shows uncond./cond. FID and, when provided, corresponding training iterations in parentheses. Baseline numbers from (Zhou et al., 2024a; Zheng & Yang, 2025).
Table 5: Conditional ImageNet 64x results.
Mothod Training From Scratch
(iters) Proc.
Roc. 1
Method
NFE
FID.U.
1S
FID.C
1000
DDPM
250
11.00
0.67
0.58
Training From Scratch
ADM
250
2.07
0.74
0.63
EDM
79
2.64
DDPM
1000
3.17
0.70
0.63
DDIM
100
4.16
0.63
Score SDE
2000
2:20
3
48
iCT-deop
3.25
0.72
0.63
iCT-ddeep
2
2.77
0.74
0.62
EDM
35
1.98
1.79
BigGAN
14.73
BigGAN deep
4,06
0.79
0.48
StyleGAN--TT.
StyleGANZ-ADA
2.92
9.82
2.42
SAN
Post-training
ICT
2
2.46
9.80
iCT-doop
1
2.51
9,76
BOOT
16.3
0.68
0.36
iCT-deop
2
2.24
9.89
DENO
1
7,83
0.61
CD
6.20
0.68
0.63
Post-training
CTM
0.70
0.57
PD
9.12
CTM
2
1.73
0.64
0.57
DMD
2.62
CD
I
3.55
sCDSS
2.97
CD
2
2.93
sCD-S
CTM
I
1.98
1,73
DMD2
1.51
CTM
1.63
MSD (DM)
2.37
DMD
MSD (ADM)
1.20
D20-F
1.54
10,10
1.44
D20-F
1.71
SID A
3,50)30K)
10.19
(540(50K)
SID
SID
(201)
1.52
0.75
0.74
0.63
0.62
SID NCE (M=1)
(35K)
10.20
1.46
NCE (X-1)
(,28(15K)
SID NCCE (X-50)
(20K)
10.23
1.39 (20K)
NºCE (K=50)
1.11(10K)
4.3
OFFLINE BLACK-BOL OPTIMIZATION
-5
(a) GT
(b) LM=t
(c) LM=10
(d) 1ar=100
-15
244
(c) DDOM (f) G-SV MLE-LD (h) MLE-SV
Figure: 3: Viz. of Branin optimal samples. (b-d) d) are results of our method. G-SV denotes the Gaussian prior model sampled with SVGD. MLE-LD and MLE-SV de- the model trained by MI. sampled with LDD and Stein Variational Gradient Descent (SVGD), respectively.
X1
Figure 2: Branin function sets. sets.
Beyond image modeling, we also explore the broader impact of our proposed technique through the lens of offline Black-Box Optimization (BBO). This task evaluates not only how well model captures the training distribution, but also its ability to internalize structural regularities and generalize to unseen queries. In the offline BBO setting (Trabucco et al., 2022), one is given a pre-collected dataset D {(x,, y:) }:) of inputs and their black-box function values h((,)) y/- At test time, the optimizer may access a limited budget Q evaluations of the unknown function and must return candidates with high observed values. This setup is particularly well-suited for assessing generative approaches to optimization, where the function value acts as a conditioning signal (Brookes et al., 2019; Kumar & Levine, 2020; K Krishnamoorthy al., 2023).
8

---

Published as a conference paper at ICLR 2026
We instantiate this view by parameterizing conditional model (x y) via latent variables z, using
(x y) o (ptx))) z)]
p(z v) x (3 Pa)
and employing stochastic samplers such as Langevin Dynamics (LD) or Stein Variational Gradient Descent (SVGD) (Liu & Wang, 2016) for conditional sampling. To train this model, we use VAE with an LEBM prior that jointly models (x, v). Full problem statement, implementation details, and extended results are deferred Appx. D.5.
4.3.1 2D BRANIN FUNCTION
We begin with the 2D Branin function (Fig. 2) as a proof of concept. This task serves as a sanity check to validate that our method can both faithfully capture high-value modes and generalize beyond the best points observed in the offline dataset (see Appx. D.6 for de- tails). Following DDOM (Krishnamoorthy et al., 2023), we perform a held-out offline optimization experiment. Specifically, we uni- formly sample N = 5000 points from the do- main [-5, 10] x [o, 15] to construct the offline dataset, then remove the top 10% by function value. This ensures that high-value contours retained, but the exact global optima are excluded during training. For evaluation, we adopt a query budget Q 128 and use SVGD for sampling. Table 6 shows that our method generalizes well beyond the maximum value in the dataset maxx approaching the true optimum and outperforming gradient-basee GA as well as modern generative baselines BONET (Mashkaria et al., 2023) and DDOM (Krishnamoorthy et al., 2023).
Table 6: Results on the top--oe-ttil--rmooved Branin task (avg. over 5 runs), OPT denotes the global optimum.
D POt.
GA
BONET
DDOM
Ours
6..1/044
4.0244
-18
-1.6g..1
0. 0.4
(a) w/ top-10% points
(b) w/o top-10% points
Figure 4: Results on uniformly sampled Branin w/ and w/o top-10% points. Zoom in for details.
We further visualize conditional samples corresponding to high function values (Fig. 3), and examine the correlation between target conditions (desired function values) and achieved function values (Fig. 4). For visualization, we draw 2560 samples from each model. In Fig. 3a, the distribution of true optima is emulated with a 3-component GMM centered at the global minima. As shown in the first row of Fig. 3, increasing M in Eq. (5) produces samples that better approximate the ground-truth distribution, while models trained with MCMC-base based MLE or vanilla NCE (M-1) 1) lag behind. This again supports our analysis that a sufficiently large M is critical for accurate gradient approximation.
Table 7: Normalized results on design- bench with Q 256 (avg, over 5 runs).
BASELINE
TFRINDS
TEBINDIO
CHEMBL
SUPERCON.
ANT
D'KITTY
MEAN SCORE'
RANK
29 (ba))
0.630
0.407
0.905
0.309
0.505
B.884
GP-EEI
0.824 0.86
06355+0.011
0.633
8.000
0.501 :: 0.021
0.887 0.000
0,806 = 0.000
0.729 4: 0.001
7.8
CMA-ES
0,333 0005
0.679 0.054
0.636
8.006
0.491
2: 0.0044
1.496
+ 0.9928
I.P25
= 0.002
0.815: 2: 0.108
5.8
REINFORCE
0.059 + 0015
0.80 0.028
0.630
0.005
0.481
+ 0.017
0.261 + 0.042
0:476 0.202
0.575 1 0.054
8.5
Gradicat Asceat
0.361 #: 0.055
0.609 0.030
0J647
11.021
0.304 =: 0.0N
0.840 = 0.004
0,506 = 00017
0.672 4: 0.021
5.0
COMs
0.964 + 0.200
0.851 0.020
0.616
0.005
0.425
+: 0.033
0.919 + 0.021
0:948 + 0.006
0.781 +: 0.018
5.8
BONET
0.975 + 0.001
0.681 + 0.035
0.654
- 0.00
0.437 + 0.0:2
0.976 + 0.012
0.964 + 0.012
0.780 + 0.023
3-2
CLAS
0.358 2: 0.018
0.607 0.017
0.600 = 11,005
0.460 2: 0.0N5
0.876 = 0.025
B1806 - 0U016
0.745 4: 0.1KU
7.8
MING
0.008 0.47
0.859 0.041
0.653
0.002
0-484 + 0.017
0.962
0.0%8
0.946 + 0.009
0.770 + 0.073
5.5
DROM
0.371 0.005
0.685 0.002
0.033
0.007
0.560 + 0.044
0.957 + 0.002
0.520 + 0.009
0.787 + 0.054
4.5
Ouix
0 0.900 - 0. 003
0.803 0.085
0.681 + 0.025
0.567 + 0.017
0.982 + 0.012
0.961
0.008
0.827 + 0.021
1.2
4.3.2 OFFLINE BBO ON DESIGN-BENCH
Task setup and evaluation. We next evaluate on gigher-dimensional real-world tasks' from Trabucco et al. (2022), grouped into discrete optimization (TF-Bind-8, TF-Bind-10, ChEMBL. and continuous optimization (D' Kitty, Ant Morphology, Superconductor). We compare against three categorics of baselines: (i) generative inverse models with different parameterizations, including CbAS (Brookes et al., 2019), CbAS (Fannjiang & Listgarten, 2020), MIN (Kumar & Levine,
NAS and Hopper tasks are excluded following Mashkaria al. (2023); see Appx. D.7.2.

---

Published as a conference paper at ICLR 2026
Table 8: Ablation studies design-bench (avg, over 5 runs), results with budget 256.
BASELINE
TFBINDS
TFBIND10
CHEMBL
SUPERCON.
ANT
D'KITTY
MEAN SCORE'
D (bess)
0.439
0.457
0.309
0.565
0,881
0.805
sumpler
FRM
0.524 +
0.000
0,33
0.006
0,622 + 0.049
- 0.035
N'CE-LERM (KW100, Kw6)
0.073
0,7.30 0.086
0,009
0.013
0.363 0.006
0,857
0.005 0.004
0.748 0.025
SVGI sumpler
FRM
0.941 0.001
0.681
0.079
0.634 0.002
0.286
0.004
0.926 0.016
0.916 0.007
0.732 + 0.014
NCE LEBM (H 62
0.961
0.004
0.635 1. 0.034
0.640 1: 0.011
0.497 0.004
0.907 1. 0.007
0.959 0.002
0.7710 2: 0.007
N-CF-LFFM (00100,60)
0.722 0.043
0.948 + 0.005
0:421 0.021
0.803 + 0.008
0.053 + 0.008
0.764 0.005
N'CE-LEMM (K-100, K=6) 0,990 + 003
0.803 0.086
0.681 + 0.026
0.567 + 0.017
0.962 0.012
0.961 + 0.006
0.827 + 0.002
2020), and DDOM (Krishnamoorthy et al., 2023); (ii) gradient(-like) updating from existing designs, such as gradient-ascent- aaade---eeetteeed methods (Fu & Levine, 2020; Yu et al., 2021b; Trabucco et al., 2021; Chen al., 2022; Qi et al., 2022; Yuan et al., 2024; Chen et al., 2024) and BONET (Mashkaria et al., 2023); (iii) additional baselines including REINFORCE and evolutionary algorithm reported in Trabucco al. (2022). Further details are given in Appx. D.7.1 and D.7.3.
Following standard practice (Trabucco et al., 2022), we report normalized ground-truth function values yn. y --iiai where Vinin and yenax the minimum and maximum values in the full Minax - Viiin
dataset (unseen during training). We summarize performance using both mean scores and mean normalized ranks (MNR) across tasks. We provide two sets of results: (1) Q 256 queries following Krishnamoorthy et al. (2023); (2) Q = 128 queries following Trabucco et al. (2021); Chen et al. (2024), excluding ChEMBL. In the main text we report 100-th percentile results with Q 256, and defer Q- = 128 results and additional percentiles (e.g., 50-th) to Appx. D.7.4.
Main results. Tables 7 and 23 show that method attains the best average ranks of 1.2 (Q 256) and 1.8 (Q = 128), compared to runner-ups BONET (Mashkaria et al., 2023) and Tri-mentoring (Chen al., 2024), which achieve 3.7 and 2.8, respectively. Our approach consistently yields the highest normalized mean values across tasks, outperforming all baselines baselines-particulary generative inverse models-and confirming the effectiveness of our design. With Q 256, we obtain the best performance on five of six tasks, with the sole exception being ANT. Notably, the best west-performing baseline on this task, CMA-ES, exhibits a much higher variance (std. 0.928) compared to ours (0.012), underscoring its instability to initialization. With Q = 128, we achieve the best results on three out of five tasks. While ExPT (Nguyen al., 2024) achieves competitive results on the remaining tasks, it heavily relies on pretraining from larger datasets and is not directly comparable to other baselines. Nevertheless, our method delivers top-tier performance across all tasks.
In Table 8, we further compare method with MLE-LEBM and NCE (with ratio decomposition) under both LD and SVGD sampling. We find that the vanilla MLE paradigm, and even NCE, perform poorly in the offline BBO setting-like likelyy due to the complex, highly multimodal joint latent space of (x, y). By contrast, our N2CE- model produces strong results even with basic LD sampling, in line with our analysis on the importance of noise magnitude. Extended results, including ablations over M, K (Tab. 26) and robustness to query budget Q (Tab. 27), are deferred to Appx. D.7.4.
CONCLUSION
In this work, we revisited NCE from the perspective of noise magnitude, an often-overlooked factor in ratio estimation. Building on this view, we introduced the NºCE framework, interpreting "noisier" NCE objective as a controlled upproximation MLE and providing both theoretical and practical insights into its behavior. Our analysis shows that, under mild conditions, the CºCE gradient aligns with the MIE gradient when the noise magnitude is sufficiently large, and we further examined the simite-sample regime to motivate simple yet effective regularizations.
Empirically, CCCE delivers consistent improvements across image modeling, anomaly detection, and offline black-box optimization, outperforming strong baselines and demonstrating its versatility, More broadly, our results suggest that NºCE provides a practical and theoretically grounded framework for learning rewards and critics. Looking ahead, see particular promise in extending this framework to discrete domains such as language modeling and to multimodal tasks like eoxt-to-image and text-to-vides generation, where traditional NCE remains limited.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
Y. W. is partially supported by NSF 2415226,, DARPA W912CG25CA007 and research gift funds from Amazon and Qualcomm. We gratefully acknowledge Lambda, Inc. for providing the computational resources used in this project. We would also like to express our gratitude to the anonymous reviewer PhJz, for drawing our attention the connection with NWJ and for contributing directly to Sec. 3.4. We also sincerely thank the anonymous AC FyXy for the careful handling of our submission and for the time and attention devoted to the review process.

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We provide complete statements and proofs of all theoretical results in Appx. B. Implementation details, including key hyperparameters, network architectures, and step-by-step training instructions, are described in Appx. C. Comprehensive descriptions of our experimental setups, datasets, and evaluation metries are available Appx. D. We will release code, data, and model checkpoints upon acceptance of this manuscript to ensure full reproducibility.
12

---

Published as a conference paper at ICLR 2026
REFERENCES
Jyoti Aneja, Alex Schwing, Jan Kautz, and Arash Vahdat. A contrastive learning approach for training variational autoencoder priors. Advances in neural information processing systems, 34:480-44 493, 2021. 20
Christof Angermueller, David Dohan, David Belanger, Ramya Deshpande, Kevin Murphy, and Lucy Colwell. Model-based reinforcement learning for biological sequence design. In International conference on learning representations, 2019.
Christof Angermueller, David Belanger, Andreea Gane, Zelda Mariet, David Dohan, Kevin Murphy, Lucy Colwell, and D Sculley. Population-based black-box optimization for biological sequence design. In International Conference Machine Learning, pp. 324-334. PMLR, 2020. 21
Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Neural photo editing with introspective adversarial networks. arXiv preprint UrXiv:1609.07093, 2016. 29
David Brookes, Hahnbeom Park, and Jennifer Listgarten. Conditioning by adaptive sampling for robust design. In International conference on machine learning, pp. 773-782. PMLR, 2019. 8, 9, 21, 36, 37
Ciwan Ceylan and Michael U Gutmann. Conditional noise-contrastive estimation of unnormalised models. In International Conference Machine Learning, pp. 726-734. PMLR, 2018. 20
Can Chen, Yingxueff Zhang, Jie Fu, Xue Steve Liu, and Mark Coates. Bidirectional learning for offline infinite-width width model-based optimization. Advances in Neural Information Processing Systems, 35:29454-29467, 2022. 10, 27,37
Can Sam Chen, Christopher Beckham, Zixuan Liu, Xue Steve Liu, and Chris Pal. Parallel-mentoriig for offline model-based optimization. Advances in Neural Information Processing Systems, 36, 2024. 10, 21, 37, 39
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp- 1597-1607. PMLR, 2020. 1
Nando de Freitas and Ziyu Wang. Bayesian optimization in high dimensions via random embeddings. 2013. 20
Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-based models. arXiv preprint arXiv:1908 190,0099,,,,,.9, 2019. 2, 20,
Yilun Du, Shuang Li, Joshua Tenenbaum, and Igor Mordatch. Improved contrastive divergence training of energy based models. In International Conference Machine Learning (ICML), 2021. 6, 20
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and
stochastic optimization. Journal of machine learning research, 12(7), 2011. 30
Clara Fannjiang and Jennifer Listgarten. Autofocused oracles for model-based design. Advances in Neural Information Processing Systems, 13:12245-12956 2020. 9, 21, 36
Justin Fu and Sergey Levine. Offline model-based optimization via normalized maximum likelihood estimation. In International Conference Leurning Representations, 2020. 10, 21, 36, 37
Ruiqi Gao, Erik Nijkamp, Diederik P Kingma, Zhen Xu, Andrew M Dai, and Ying Nian Wu. Flow contrastive estimation of energy-based models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7518-7528, 2020a. 20
Ruiqi Gao, Yang Song, Ben Poole, Ying Nian Wu, and Diederik P Kingma. Learning energy based models by diffusion recovery likelihood. In International Conference on Learning Representations (ICLR), 2020b. 20
13

---

Published as a conference paper at ICLR 2026
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Tch, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International conference machine learning, pp- 1 7713- PMLR, 2018a. 21
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint 1rXiv:1807.01622 2018b. 21
Garnett, M Oshorne, and Hennig. Active learning of linear embeddings for gaussian processes. In 30th Conference on Uncertainty in Artificial Intelligence (UAI 2014). pp. 230-239. AUAI Press, 2014. 20
Rafael Bombar-oom,,,lli,, Jennifer N Wei, David Duvenaud, José Miguel Bermández-Lobard Benjamín Sánchez-I Sánchez-Lengelingg Lengeling, Dennis Sheberla, Jorge mauilera-pparrguirree Timothy D Hirzel, Ryan P Adams, and Alán Aspuru-Guzik, Automatic chemical design using data-driven continuous representation of molecules. ACS central science, 4(2):268-22 276, 2018. 20
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11))139-144, 2020. 21, 37
Aditya Grover and Stefano Ermon. Boosted generative models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018. 20
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 297-304. JMLR Workshop and Conference Proceedings, 2010. 1,2
Michael U Gutmann and Aapo Hyvärinen. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. Journal of machine learning research, 13(2),
2012. 1, 2, 3, 4, 20
Tian Han, Yang Lu, Song-Chun Zhu, and Ying Nian Wu. Alternating propaproppgttion for generator network. In AAAI Conference on Artificial Intelligence (AAAT), 2017. 33
Tian Han, Erik Nijkamp, Linqi Zhou, Bo Pang. Song-Chun Zhu, and Ying Nian Wu. Joint training of variational auto-encoder and latent energy -baed model. In Conference Computer Vision and Pattern Recognition (CVPR), 2020. 33
Nikolaus Hansen. The ema evolution strategy: a comparing review. Towards a new evolutionary computation: Advances in the estimation of distribution algorithms, pp. 75-102, 2006. 37
Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The elements of statistical learning: data mining, inference, and prediction, volume Springer, 2009. 20
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in Neural Information Processing Systems (NeurIPS), 2017. 7
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, In Advances in Neural Information Processing Systems (NeurIPS), 2020. 1, 21, 26, 29,37 37
Yiwen Huang, Aaron Gokaslan, Volodymyr Kuleshov, and James Tompkin. The gan is dead; long live the gan! modern gan baseline. arXiv preprint arXiv:2501. 01.5501..544,,20 2025. 1
Kirthevasan Kandasamy, Jeff Schneider, and Barnabás P6ezos. High dimensional bayesian opti- misation and bandits via additive models. In International conference on machine learning, pp. 295-304. PMLR, 2015. 20
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 4401-4410 2019.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in neural information processing systems, 04226665-26577 2022. 1
14

---

Published as a conference paper at ICLR 2026
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Gamelo,, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. In International Conference on Learning Representations, 2018. 21
Minsu Kim, Federico Berto, Sungsoo Ahn, and Jinkyoo Park. Bootstrapped training of score- conditioned generator for offline design of biological sequences. Advances in Neural Information Processing Systems, 36, 2024. 21, 37,
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint TrXiv:1412.6988 2014. 28, 29, 30
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.61 14, 2013. 33
Durk P Kingma and Prafulla Dhariwal. Glow:
Generative flow with invertible 1xl convolutions.
Advances in neural information processing systems, 31, 2018. 20
Deqian Kong, Yuhao Huang, Jianwen Xic, and Ying Nian Wu. Molecule design by latent prompt transformer. arXiv preprint arXiv:2310.03353, 2023a. 20, 34
Deqian Kong, Bo Pang, Tian Han, and Ying Nian Wu. Molecule design by latent space energy-based modeling and gradual distribution shifting. In Uncertainty in Artificial Intelligence, pp. 1109-1120. PMLR, 2023b. 20, 34
Siddarth Krishnamoorthy, Satvik Mehul Mashkaria, and Aditya Grover. Diffusion models for black- box optimization. In International Conference on Machine Learning, 2023. 8, 9, 10, 20, 21, 34, 35, 36, 37, 38
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
Aviral Kumar and Sergey Levine. Model inversion networks for model-based optimization. Advances in Neural Information Processing Systems, 33:5126-51333 2020. 8, 9, 21, 35, 36, 37
Rithesh Kumar, Sherjil Ozair, Anirudh Goyal, Aaron Courville, and Yoshua Bengio. Maximum entropy generators for energy-based models. arXiv preprint arXiv: 0850000,00000,,,,0.0000000080008 2019.
Matt J Kusner, Brooks Paige, and José Miguel Hernández- Lobato.. Grammar variational autoencoder. In International conference on machine learning. pp. 1945-1954. PMLR, 2017. 20
Justin Lazarow, Long Jin, and Zhuowen Tu. Introspective neural networks for generative modeling. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2774-2783, 2017.
20
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exil/mnist/, 1998.
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, Fujie Huang, et al. A tutorial on energy-based learning. Predicting structured data, 1(0). 2006. 2, 3
Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2020.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamy Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02247, 2022.
Bingbin Liu, Elan Rosenfeld, Pradeep Kumar Ravikumar, and Andrej Risteski. Analyzing and
improving the optimization landscape of noise-contrastive estimation. In International Conference
on Learning Representations, 2021. 2, 3, 4, 24, 25
Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose bayesian inference algorithm. Advances neural information processing systems, 29, 2016. 9, 30, 34
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.00003, 2022.
15

---

Published as a conference paper at ICLR 2026
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015. 6
Xiaoyu Lu, Javier Gonzalez, Zhenwen Dai, and Neil D Lawrence. Structured variationally auto- encoded optimization. In International conference on machine learning, pp. 3267-3275. PMLR, 2018. 20
Weijian Luo, Tianyang Hu, Shifeng Zhang, Jiacheng Sun, Zhenguo Li, and Zhihua Zhang. Diff- instruct: A universal approach for transferring knowledge from pre-trained diffusion models. Advances in Neural Information Processing Systems, 36:76525-76546, 2023. 20
Satvik Mehul Mashkaria, Siddarth Krishnamoorthy, and Aditya Grover. Generative pretraining for black-box optimization. In International Conference on Machine Learning, pp. 24173-24197.
PMLR, 2023. 10, 20, 21, 35, 36, 37, 38, 39
David McAllester and Karl Stratos. Formal limitations on the measurement of mutual information. In International Conference on Artificial Intelligence and Statistics, pp. 875-884, PMLR, 2020.
Aditya Menon and Cheng Soon Ong. Linking losses for density ratio and class- bass-probabiilty estimation. In International Conference on Machine Learning, pp. 304-313. PMLR, 2016. 3
Azalia Mirhoseini, Anna Goldie, Mustafa Yazgan, Joe Jiang, Ebrahim Songhori, Shen Wang, Young- Joon Lee, Eric Johnson, Omkar Pathak, Sungmin Bae, al. Chip placement with deep reinforce- ment learning. arXiv preprint arXiv:2004.10746 2020. 21
Mchdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv:1411.174. 2014. 27,
arXiv preprint
A Mnih and Teh. A fast and simple algorithm for training neural probabilistic language models. In International Conference Machine Learning (ICML), volume 2, 2012. 4, 27, 29
Riccardo Moriconi, Marc Peter Deisenroth, and KS Sesh Kumar. High-dimensione bayesian optimization using low-dimensional feature spaces. Machine Learning, 109:1925-1999 2020aa 20
2020a.
Riccardo Moriconi, KS Sesh Kumar, and Marc Peter Deisenroth. High-dimensional bayesian optimization with projections using quantile gaussian processes. Optimization Letters, 14:51-64, 2020b. 20
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Tung Nguyen, Sudhanshu Agrawal, and Aditya Grover. Expt: Synthetic pretraining for few-shot experimental design. Advances in Neural Information Processing Systems, 36, 2024. 10, 37, 39
Vu Nguyen and Michael A Osborne. Knowing the what but not the where in bayesian optimization. In International Conference on Machine Learning, pp. 7317-7326. PMLR, 2020. 21
XuanLong Nguyen, Martin J Wainwright, and Michael Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56(11):5847-581,, 2010. 3.
Erik Nijkamp. Mitch Hill, Song- Cnng-Chun Zhu, and Ying Nian Wu. Learning non-convergent non- persistent short-run meme toward energy-based model. In Advances in Neural Information Processing Systems (NeurIPS), 2019. 20
Erik Nijkamp, Mitch Hill, Tian Han, Song-Chun Zhu, and Ying Nian Wu. On the anatomy of meme-based maximum likelihood learning of energy-based models. In AAAI Conference Artificial Intelligence (AAAI), 2020. 3, 20
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. Advances in neural information processing systems, 29, 2016. 5
16

---

Published as a conference paper at ICLR 2026
Bo Pang, Tian Han, Erik Nijkamp, Song-Chun Zhu, and Ying Nian Wu. Learning latent space energy-based prior model. In Advances in Neural Information Processing Systems (NeurIPS), 2020a. 6, 7, 20, 32, 33
Bo Pang, Tian Han, and Ying Nian Wu. Learning latent space energy-based prior model for molecule generation. arXiv preprint arXiv:2010.09351, 2020b. 20
Bo Pang, Tianyang Zhao, Xu Xie, and Ying Nian Wu. Trajectory prediction with latent belief energy-based model. In Conference on Computer Vision and Puttern Recognition (CVPR), 2021.
20
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), 2019. 28, 29
Jan Peters and Stefan Schaal. Reinforcement learning by reward- -eightte regression for operational space control. In Proceedings of the 24th international conference on Machine learning, pp. 745-750, 2007.
Miika Pihlaja, Michael Gutmann, and Aapo Hyvärinen. A family of computationally efficient and simple estimators for unnormalized statistical models. In Proceedings of the Twenty-Sixth Conference on Uncertainly in Artificial Intelligence, pp. 442-449, 2010.
Yury Polyanskiy and Yihong Wu. Information theory: From coding to learning. Cambridge university press, 2025. 5
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pp. 5171-5180. PMLR, 2019. 3
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d
diffusion. arXiv preprint arXiv:2209. 14988, 2022. 20
Han Qi, Yi Su, Aviral Kumar, and Sergey Levine. Data-driven offline decision-making via invariant representation learning. Advances Neural Information Processing Systems, 35:13226-13237, 2022. 10, 21, 37
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015. 26
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 9748-8763 PMLR, 2021.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning (ICML), 2014. 20
Benjamin Rhodes, Kai Xu, and Michael U Gutmann. Telescoping density-ratio estimation. Advances in neural information processing systems, 33:4905-4916, 2020. 1, 3, 5, 20
Paul Rolland, Jonathan Scarlett, Ilija Bogunovic, and Volkan Cevher. High-dimensional bayesian optimization via additive models with overlapping groups. In International conference on artificial intelligence and statistics, pp. 298-307. PMLR, 2018. 20
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Reuven Rubinstein. Optimization of computer simulation models with rare events. European Journal of Operational Research, 99(1):99-112 1997. 21
17

---

Published as a conference paper at ICLR 2026
Reuven Y Rubinstein and Dirk P Kroese. The cross-entropy method: unified approach to combina- torial optimization, Monte-Carlo simulation, and machine learning, volume 133. Springer, 2004. 21
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512, 2022. 20
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1):
148-175, 175, 2015. 20, 34
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25, 2012. 21, 34
Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Mostofa Patwary, Mr Prabhat, and Ryan Adams. Scalable bayesian optimization using deep neural networks. In International conference on machine learning, pp. 2171-2180. PMLR, 2015. 21
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. -csre-based generative modeling through stochastic differential equations. In International Conference on Learning Representations (ICLR), 2020. 1
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In International Conference on Machine Learning, pp. 32211-32252. PMLR, 2023. 20
Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process opti-
mization in the bandit setting: regret and experimental design. In Proceedings of the 27th International Conference on International Conference on Machine Learning, pp. 1015-1022, 2010.
21
Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density ratio estimation in machine learning. Cambridge University Press, 2012. 3, 20
Jakub Tomezak and Max Welling. Vae with a vampprior. In International Conference on Artificial Intelligence and Statistics, pp. 1214-1223. PMLR, 2018. 26
Brandon Trabucco, Aviral Kumar, Xinyang Geng, and Sergey Levine. Conservative objective models for effective offline model-based optimization. In International Conference on Machine Learning, pp. 10358-10368. PMLR, 2021. 10, 21, 37
Brandon Trabucco, Xinyang Geng, Aviral Kumar, and Sergey Levine. Design-bench: Benchmarks for data-driven offline model-based optimization. In International Conference on Machine Learning, pp. 21658-21676. PMLR, 2022. 8, 9, 10, 21, 34, 36, 37
Zhuowen Tu. Learning generative models via discriminative approaches. In 2007 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-8. IEEE, 2007. 20
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv: 2016. 28
Zhendong Wang, Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Diffusion-gan: Training gans with diffusion. arXiv preprint arXiv:220.00266,, 2022. 20
Zhengyi Wang, Cheng Lu. Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lifiedreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in neural information processing systems, 36:8406-8441, 2023. 20
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In International Conference on Machine Learning (ICML), 2011. 3
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229-256, 1992. 21, 37
James T Wilson, Riccardo Moriconi, Frank Hutter, and Mare Peter Deisenroth. The reparameterizztion
trick for acquisition functions. arxiv preprint arXiv:/712.00442, 2017. 37
18

---

Published as a conference paper at ICLR 2026
Zhisheng Xiao and Tian Han. Adaptive multi- density ratio estimation for learning latent space energy-based model. Advances in Neural Information Processing Systems (NeurIPS), 2022. 3, 33
Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu. A theory of generative convnet. In International Conference Machine Learning, pp. 2635-2644, 2016. 2,
Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6613-6623, 2024. 20
Sangwoong Yoon, Yung-Kyun Noh, and Frank Park. Autoencoding under normalization constraints. In International Conference on Machine Learning, pp. 12087-12097 PMLR, 2021. 33
Sangwoong Yoon, Young-Uk Jin, Yung-Kyun Noh, and Frank Park. Energy-based models for
anomaly detection: A manifold diffusion recovery approach. Advances in Neural Information Processing Systems, 36:49445-49466, 2023. 6, 32,
Sangwoong Yoon, Himchan Hwang, Dohyun Kwon, Yung- Noh, and Frank Park. Maximum entropy inverse reinforcement learning of diffusion models with energy-based models. Advances in Neural Information Processing Systems, 37:24601-24624 2024. 7, 20, 29, 33
Peiyu Yu, Sirui Xie, Xiaojian Ma, Yixin Zhu, Ying Nian Wu, and Song-Chun Zhu. Unsupervised foreground extraction via deep region competition. Advances in Neural Information Processing Systems, 34: 14264-14279, 2021a. 20
Peiyu Yu, Sirui Xie, Xiaojian Ma, Baoxiong Jia, Bo Pang, Ruiqi Gao, Yixin Zhu, Song-Chun Zhu, and Ying Nian Wu. Latent diffusion energy- based model for interpretable text modeling. arXiv preprint arXiv:2206.0588 05895, 2022. 6, 20, 26, 27, 29
Peiyu Yu, Yaxuan Zhu, Sirui Xie, Xiaojian Shawn Ma, Ruiqi Gao. Song-Chun Zhu, and Ying Nian Wu. Learning energy-based prior model with diffusion-amortized memc. Advances in Neural Information Processing Systems, 36, 2024. 6, 7, 20, 27, 32, 33
Sihyun Yu, Sungsoo Ahn, Le Song, and Jinwoo Shin. Roma: Robust model adaptation for offline model-based optimization. Advances in Neural Information Processing Systems, 34:4619-4631, 2021b. 10, 21, 37
Ye Yuan, Can Sam Chen, Zixuan Liu, Willie Neiswanger, and Xue Steve Liu. Importance-aware co-teaching for offline model-based optimization. Advances in Neural Information Processing Systems, 36, 2024. 10, 21, 39
Houssam Zenati, Chuan Sheng Foo, Bruno Lecouat, Gaurav Manek, and Vijay Ramaseshan Chan- drasekhar. Efficient gan-based anomaly detection. Proceedings of the 20th IEEE International Conference on Data Mining (ICDM), 2018, 2018. 32,
Bowen Zheng and Tianming Yang. Revisiting diffusion models: From generative pre-training to one-step generation. arXiv preprint arXiv:2506. S.XY:22506.09776, 2025.
Mingyuan Zhou, Huangjie Zheng, Yi Gu, Zhendong Wang, and Hai Huang. Adversarial score identity distillation: Rapidly surpassing the teacher in one step. arXiv preprint arXiv:2410.1499 14919, 2024a. 7, 8, 20, 29, 33
Mingyuan Zhou, Huangjie Zheng, Zhendong Wang. Mingzhang Yin, and Hai Huang. Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation. In Forty-first International Conference on Machine Learning, 2024b. 20
19

---

Published as a conference paper at ICLR 2026
A RELATED WORK
Noise contrastive estimation Noise Contrastive Estimation (NCE) (Hastie et al., 2009; Sugiyama et al., 2012; Gutmann & Hyvärinen, 2012) provides a handy interface bridging the gap between discriminative and generative learning, An interesting line of research has been devoted to estimating unnormalized density, i.e., Energy-Bass -ased Models (EBMs) or LEBMs, with NCE-liK- like objectives (Gutmann & Hyvärinen, 2012: Tu, 2007: Lazarow al., 2017: Ceylan & Gutmann, 2018; Grover & Ermon, 2018; Gao et al., 2020a; Aneja et al., 2021). Particularly, Gao et al. (2020a) recruits a normalizing flow (Rezende et al., 2014; Kingma & Dhariwal, 2018) as the base distribution for density estimation. Aneja et al. (2021) refine the prior distribution of pre-trained powerful VAE with noise contrastive learning. However, vanilla NCE, even with multi-stage estimation (Rhodes et al., 2020), may demonstrate less desirable results due to large gaps between target and noise distributions, dubbed as density-chasm. In this paper, we propose to study this fundamental problem from an often-neglected perspective: the magnitude of the noise distribution used for ratio estimation. Specifically, with an increased magnitude of the noise distribution, the gradient of the NCE objective can closely match the MLE gradient. We provide theoretical and empirical analysis showing the effectiveness of our proposed framework.
Energy-based prior model As interesting branch of EBMs (Xie et al., 2016; Nijkamp et al., 2019; 2020; Du & Mordatch, 2019; Du et al., 2021), Pang et al. (2020a) show that an EBM can serve as an informative prior model in the latent space, i.e., LEBM: such a prior greatly improves the model expressivity over those with non non-informative gaussian priors and brings strong performance on downstream tasks (Yu et al., 2021a; 2022; Pang et al., 2020b; 2021). However, learning LEBM requires MCMC sampling to estimate the learning gradients. which needs careful tuning and numer- ous iterations to converge when the target distributions are high-dimensional or highly multimodal. Commonly-usee short-run MCMC (Nijkamp et al.. 2019) in practice can lead to malformed energy landscapes and instability in training (Yu et al., 2022: 2024; Gao et al., 2020b; Nijkamp et al., 2020; Du & Mordatch, 2019; Du et al., 2021). In this work, we consider inducing the informative latent space with CºCE integrated into variational learning; the proposed method requires no MCMC for estimating the prior model, and shows reliable sampling quality in practice.
Diffusion distillation Diffusion distillation has recently emerged as a key direction for accelerating sampling while retaining the strong generative performance of diffusion models. Early efforts such as consistency models (Song et al., 2023) and progressive distillation (Salimans & Ho, 2022) demonstrated that multi-step denoising trajectories could be compressed into few steps, but these methods often faced tradc-offss between speed and fidelity. More recent approaches refine this idea by explicitly distilling score information from pretrained diffusion models. For instance, Score Distillation Sampling (SDS) and its variants (Poole et al., 2022; Wang et al., 2023; Luo et al., 2023) leverage gradients of the diffused KL divergence to align student and teacher scores at different noise levels. Extensions such as Diffusion-GAN (Wang et al., 2022) and Distribution Matching Distillation (DMD) (Yin et al., 2024) incorporate adversarial or citeriuution-mmacciin objectives to better handle mismatched supports in high- -ithenninniioaal spaces. More recently. Zhou et al. (2024b) introduced Score identity Distillation (SiD), a data-free, diveeerrddeereeee--bbe based method that enables one-step generation, later improved by SiDA (Zhou al., 2024a), which integrates adversarial loss to surpass the teacher's performance. Complementary to these, reinforcement learning- based perspectives have been proposed: Yoon et al. (2024) formulated diffusion distillation as a maximum entropy inverse reinforcement learning problem. jointly training diffusion models and energy-based models, and further developed dynamic programming techniques to enable efficient updates. Collectively, these works highlight a growing trend toward principled frameworks that unify score matching, adversarial training, and control-theoretic formulations to accelerate and strengthen diffusion generation. Our work adds to this line by revisiting noise-contrastive estimation from the perspective of noise magnitude. In particular, we introduce NºCE, framework that connects "noisier" objectives to maximum likelihood, offering both theoretical insights and practical improvements across domains.
Online BBO An important set-up for BBO problems is the online BBO, also referred to as active BBO in many literature (Kong et al., 2023a;b: Krishnamoorthy et al., 2023; Mashkaria et al., 2023). For online BBO, the proposed method can query the black-box function with limited times during training. A probably most well-know family is Bayesian Optimization (BO), with large body of prior work in the area (Shahriari et al., 2015; Garnett et al., 2014; Moriconi et al., 2020a;b; Kandasamy al., 2015; Rolland al., 2018; de Freitas & Wang, 2013; Kusner et al., 2017; Lu al., 2018; Gómez-
20

---

Published as a conference paper at ICLR 2026
Bombarelli al., 2018; Snoek et al., 2012; Srinivas et al., 2010; Nguyen & Osborne, 2020). Typically, BO employs surrogates such as Gaussian Processes to model the underlying function, where the surrogates sequentially updated by querying the black-box function with newly proposed points from an uncertainty-ave acquisition function. Other branches include derivative free methods such as the cross-entropy method (Rubinstein & Kroese, 2004), methods derived from the REINFORCE trick (Williams, 1992; Rubinstein, 1997) and reward -eighted regression (Peters & Schaal, 2007), etc. Several practical approaches have combined these methodologies with Bayesian neural networks (Snoek et al., 2012: 2015), neural processes (Kim et al., 2018: Garnelo et al., 201 8a;b), and ensembles of learned score models (Angermueller et al., 2019; 2020; Mirhoseini et al., 2020) with extensive efforts in developing advanced querying mechanisms or better approximation of the surrogates to address the online BBO problem.
Offline BBO
Recent works have shown significant progress in the offline BBO set-up (Brookes
et al., 2019;
Fannjiang Listgarten, 2020; Kumar Levine, 2020; Krishnamoorthy et al., 2023; Kim et al., 2024; Yu et al., 2021b; Trabucco et al., 2021; Chen et al., 2022; Qi et al., 2022: Yuan et al., 2024; Mashkaria et al., 2023; Trabucco et al., 2022; Chen et al., 2024; Fu & Levine, 2020). Among them, Kumar& & Levine (2020) trains stochastic inverse mapping po (x (1) < Ep(x)) [p(x|z,,) y) with conditional GAN-like model to instantiate p( x 2. v) (Goodfellow et al., 2020; Mirza & Osindero, 2014). The method optimizes over z given the offline dataset maximum ymaxx and map z back to x for BBO solutions. The optimization process over draw approximate samples from p (g). DDOM (Krishnamoorthy et al., 2023) consider directly parameterizing the inverse mapping po (xly) with a conditional diffusion model in the input design space utilizing the expressiveness of DDPMs (Ho et al., 2020). We show in this paper that our framework demonstrates significant improvements over previous parameterizations of inverse model. Forward methods (Fu & Levine, 2020; Trabucco et al., 2021; Chen et al., 2022; Qi et al., 2022: Yuan et al., 2024; Chen et al., 2024) employ gradient ascent to optimize the learned surrogates propose candidate solutions for BBO. One critical issue, however, exists for the surrogate: the forward model can assign the out-of www.oofaiiing-iiiieee distiing-dtiibutionn input designs with erroneously large values or underestimate the true support of the distribution (Kumar & Levine, 2020; Mashkaria et al., 2023; Chen et al., 2024), especially when the set of valid inputs lies in low-dimensional manifold in the nigh-dimensional input design space. Several inspiring works have focus extensively on addressing this issue (Yu et al., 2021b; Trabucco et al., 2021; Chen et al., 2022; Qi et al., 2022; Yuan et al., 2024; Chen et al., 2024). For example, Yu et al. (2021b); Trabucco et al. (2021) assign lower scores identified outliers that the approximated surrogate is expected to be robust. Mashkaria et al. (2023) instead mimic the optimization trajectories of black-box optimizers, and rolls out evaluation trajectories from the trained sequence model for optimization. Although the method circumvents the need of the surrogate, it assumes knowledge of the approximate value of the true optima during optimization, and can struggle relative to other approaches on certain domains. Our method learns an informative latent space suitable for BBO, and delivers strong results in comparison to prior works.
21

---

Published as a conference paper at ICLR 2026
B DETAILED DERIVATIONS AND PROOFS
B.1 PROOF OF PROPOSITION 3.1
Assumption B.1 (Well-defined ratio). The reference distribution qo does not depend on and its support covers that of Pax Hence rx = Pa (x)x///// qx()) is well-defined for all x.
Assumption B.2 (Differentiabilityy and integrability). For each x, Pa (x) is differentiable in a, and the score function V log Pa (x) is integrable with respect both qa and Pac
Proof. Recall the objective
(m(a)= = E,
log
Ta M + Ta
+ MME
E
M log
M + Ta
By Assumption B.2, we may interchange Va and E[[.] For the first term,
To = = 1 M )
M
Va log Since qo does not depend on a, Va Pogra
Va log Ta.
M Ta
Ta
M +ra
= Va log Pa. Hence
Ta
M
Va Bgg
log
= Egg.
Va logPa
M +
M +ra
For the second term,
M
Ta
V log
Vata
Va log Par
M rra
M +ra
M Ta
so
M V Eggo
log
M
= M Egg
Ta
Va log
M +Ta
M + Ta
Using the change of measure Ego [rag] = Epa [g] from Assumption B.I, we get
M go
Ya
g EEp
Epa
M
9
M tra
M ra
Applying this with g = Va log Pa yields the stated gradient identity.
For the limit, note that Mf+ra pointwise M co and M+ro S By Assumption B.2, dominated convergence applies, giving
M
M
lim Eg. +00
Va log = E.g. Va log
M + Te
and similarly for the expectation under Pa- Combining finishes the proof.
22

---

Published as a conference paper at ICLR 2026
B.2 PROOF OF PROPOSITION 3.3
Proof. Since vaZu( is an unbiased estimator of Va (a)( we have the standard bias-variance decomposition:
(a - (Ry VaJal - -Ep, (afal) /
(10)
= Var (vati((a))
+ Va (x((aa - [V - Epa |Vafal)
Variance term. Since {(x), x;) ~ (g0, 9.) are i..dd
Var (VoCu((a)) = Var Va log
M (x())) + M log
(x
M
M + rx(p))
(1I)
1 Var
M
Va loggaa()
M M+ra(x)) Vara(z)))
M
M +r
Using b)2 < 2a² + 2b2, obtain
Var (Vanr(aa) - _E
M
Va logra (xx))
2
+ --
2
M
V VaTa(x))
M + (x
M + (x(h))
(12)
M Since
s Var (VaTa((a))
1 and
M for any M,r O, we further bound
M++
M+F
x 2 E
log Pogra(*) (x!))l
(13)
min
M E
|| r. (x))))
[
Thus the variance is controlled by the second moments of the score (or equivalently V VaTa), and in the typical regime where the first branch dominates, it decays as D(M2/n)
Bias term. From the proof of Proposition 3.1,
|
rx(x) M + rx(x) (9. - (xa(x)) V (x) dx
Va (a) (Eg. Salal - Ep. (afaa)
(x(xx) (g.(x) - Pa(x)) Vafa(x) dx
M
(14)
Hence the squared bias vanishes at rate O11M2)..
Conclusion. Combining equation and equation 14, the mean squared error of VaΠ satisfies
vaCan (Ey, VVfaal - Ep. V Vafal) = 00 M² + +
which completes the proof.
23

---

Published as a conference paper at ICLR 2026
B.3 PROOF OF THEOREM 3.2
As in Liu et al. (2021), we work with the exponential family
Pa (x) = exp α T 7(x) - -log 2(xz),
where a _ the natural parameter, T(x) R sufficient statisties, and ((x) is the partition function. For convenience in derivatives, we also use the extended parameterization
[a,, az), az lo (()), T(x) [T(x), -1];
so that pr (x exp(r Expectations E+ are taken w.r.t. Pr.
Assumption B.3 (Exponential-famin regularity), The map a + log Z(x) is twice continuously differentiable on the parameter ser A. Equivalently, Px(x) is C² in a and differentiation under the integral is valid for the quantities considered.
Assumption B.4 (Reference coverage and Fisher regularity). The reference qo covers the support of Pax so Ta (x) Pa(x)/ x) is well-defieed. Moreover, there exist constants Amii S Amax oo such that, for all extended parameters
I I I [T[T(x)[(x)]] I Amaxt.
Assumption B.5 (Bounded parameter set). ||a]|2 for all A
Lemma B.6 (Convexity). Let ra = Pa/qo with Pa in the exponential family. Under Assump- tions B.3 and B.4, the negative N CE objective in Eq. (5) is convex in
Proof. Use the extended parameterization pr (x) = TT())) By Assumption B.3 and finite second moments implied by Assumption B.4, may compute the derivatives inside the integral.
The gradient can be written as
M xx())
V (M((T)= (x(x) Mqx(x) (1.(x) - px())) T(x) dx.
(15)
Differentiating once more gives the Hessian
M px(x) (x)
V [M(()== (p+(x) ( Mgo(x))²
(a. (x) + Mxo(x)) T(x) dx.
(16)
For each the scalar weight
M (x (x()) (pr(x) + Mga(x))²
(xx Mgo(x)) 2 0.
and (x) 2 0. Hence the integrand equation is positive semidefinite; with the leading
minus sign, the Hessian is negative semidefinite for all r. Therefore - Las is convex in
Lemma B.7 (Condition number).
Under Assumptiion B.3 and B.4,
lim M M000
42 LM((T)) E.. [xx]T(x)T] T.
(17)
and hence
n5( lim V2 (2((T))
Anax
(18)
M-020
Amiin
Moreover, for large but finite M,
Amax
x 2 L.M(TT))
(1 O(1/()
(19)
Amin
24

---

Published as a conference paper at ICLR 2026
Proof. From equation 16, for any write
1 1+ Mg
V2 V. (M() = - (x,
dx((x))=
Pr (x()) 2
(1
Mgn(x)
Pointwise, Φw (x; T)
M oo, and 0 < <M<(x:7) 1 + O(1M)) uniformly. By
Assumption B.4, Ep. [Tx)] < oc, dominated convergence applies. Assumption B.3 ensures differentiation and expectations are well-defined. Hence
lim V CM)(()) = = [x())((x)
Ex,
M M00
Evaluating at the optimum T" yields equation 17. By Assumption B.4, the eigenvalues of E, [T())((x) lie [Amin; Amax], which gives condition-number bound equation 18.
For finite M, write
-V? (M = E.. (T())((x)'] = RM.
RM Bp.. (M
Since SM (*) O(1/M) and EE, (IT(((1 oo, we have |Ral O(1/M). Standard eigenvalue perturbation bounds then give equation 19.
Theorem B.8 (Convergence rate). Let Lm denote the NCCE objective. Under Assumptions B.3 and B.4, the Hessian of - LM satisfies the local smoothness and curvature bounds required in Theorem 5.1 of Liu al. (2021). Consequently, normalized gradient descent LM with M + oo converges to the optimum T at a rate polynomial in the condition number. In particular, for any tolerance 8 O,
==1
T
Amax
Amina
82
iterations suffice obtain some iterate Tę with ||
T- 112 S
This follows from Theorem 5.1 of Liu et al. (2021), together with convexity (Lemma B.6) and condition number bound (Lemma B.7), which verify the smoothness and curvature assumptions in that theorem.
Proof.
Remark. The NCCE loss inherits favorable convexity and well-conditioned curvature from the exponential family, ensuring that normalized gradient methods converge efficiently the optimum.
25

---

Published as a conference paper at ICLR 2026
C NETWORK ARCHITECTURE AND IMPLEMENTATION DETAILS
Code We promise to release code, data and model checkpoints upon acceptance of this manuscript.
C.1
HOw LEARN LATENT SPACE MODEL WITH NºCE
We show how the proposed form can be integrated into the variational learning scheme learn an accurate and explicit density Pa- We can first see that the target distribution for Pa is the aggregated posterior 4K+1(2) (z) = 9e (1lx)p(()dss (Tomczak & Welling, 2018), 8), where d denotes the posterior distribution parameterized by the encoder network. When considering ratio decomposition, we can form corresponding KL divergence which leads to the following ELBO:
ELBO
K
ELBOg. Egg llog p=(x) - DKL( 9k)
Σ
DKL Oku(gxtl//aa,
(20)
km0
which is valid ELBO by the non-negativity of the KL Divergence. B denotes the decoder network.
Training algorithms We directly learn the log-ratio estimator far. logTas for implementation. Let (.) denote the sigmoid function, the gradient for learning each ratio estimator (Paz 1K can be derived from Eq. (5) as:
Vas ELBO se Vos Bress llogo (Fac log m) + Vor, ME, [og (1- (Far llg
(21)
Let 3b = {B, Φ} collect the parameters of the decoder and encoder networks. For (0(z/x) and (x(z),), calculate the learning gradient follows:
K Σ LA=0
V ELBO = V Rg [log
V DKL. (pollpp)))) V
San
(22)
Alg. summarizes the training procedure.
Implementation When optimizing the ELBO in Eq. (20), we randomly choose one intermediate stage to optimize at each training iteration as in Ho et al. (2020). To construct these intermediate distributions, we follow Yu et al. (2022) to spherically interpolates between pairs of samples 2
[K+1 and Zo goi samples from the k-th intermediate distribution 9k are: Zς vI 00 0B3K+1, where the ) form increasing sequence from 0 to 1 controlling the distance between these implicitly defined distributions {qx 1==1 We use a linear schedule to specify of and
K
construct the intermediate distributions {9x ) T==
Algorithm 1 Variational Learning with NºCE
Input: dataset D
(x 1 params. (oa B.
Output: updated params. ((ai) x²)) repeat
posterior sampling: For each pair of {x()),, sample 2 ~ (0(2)x(0) using inference network.
forming {yu k= For each sample k+ sample h Unif((O,..., K}) and obtain
(mu: (K)) using 2 V o. 04ZK+1.
Update {ar ) with Eq. (21). learning //> = (B, Φ): Update &b with Eq. (22).
learning {ork) KK
until converged.
C.2 IMAGE MODELING AND ANOMALY DETECTION
Architecture We provide detailed network architecture for the log-ratio estimator for each stage fai k) in Tab. 11. For the VAE model, the decoder network has simple deconvolution structure similar to DCGAN (Radford et al., 2015) shown in Tab. 9. The encoder network to embed the observed images has fully convolutional structure, shown in Tab. 10.
26

---

Published as a conference paper at ICLR 2026
Table Network structures of the generator networks used for the SVHN, CelebA, CIFAR-IO, He MNIST (from top to bottom) datasets. ConvT(n) indicates a transposed convolutional operation with output channels. We use ngf=64 64 for the SVHN dataset and ngi=128 for the rest. LReLU indicates the Leaky-Rel.J activation function. The slope in Leaky ReLU is set to be 0.2.
Layers
Out Size
Stride
Input:
1x1x100
4x4
ConvT(ngf x 8), LReLU
4x4x x 8)
1
4x4
ConvT(ngf x 4), LReLU
8x8x x 4)
2
4x4
ConvT(ngf x 2), LReLU
16x16x(ngf
2
4x4 ConvT((3, Tanh
32x32x3
2
Layers
Out Size
Stride
Input:
1x1x100
4x4
ConvT(nng' x 8), LReLU
4x4x(ngf x 8)
4x4
ConvT(ngf x 4), LReLU
8x8x(ngf x 4)
2
4x4
ConvT(ngf x 2), LReLU
16x16x(ngf x 2)
2
4x4
ConvT(ngf x 1), LReLU
32x32x(ngf
2
4x4 ConvT((3), Tanh
64x64x3
2
Layers
Out Size
Stride
Input:
1x1x128
8x8
ConvT(nng' x 8), LReLU
8x8x(ngf x 8)
4x4
ConvT(ngf x 4), LReLU
16x16x(ngf
2
4x4
ConvT(nnf x 2), LReLU
32x32x(ngf x 2)
2
3x3 ConvT(3), Tanh
32x32x3
1
Layers
Out Size
Stride
Input:
1xIx128
4x4
Conv'T(ngf x 16), LReLU
4x4x(ngf 16)
1
4x4
ConvT(nnf x 8), LReLU
8x8x(ngf x 8)
2
4x4
ConvT(ngf x 4), LReLU
16x16x(ngf 4)
2
4x4
ConvT(nnf x 4), LReLU
32x32x(ngf x 4)
2
4x4
ConvT((nf x 2). LReLU
64x64x(ngf 2)
2
4x4
ConvT(ngf x 1), LReLU
128x128x(ngxx
2
4x4 ConvT(3), Tanh
256x256x3
2
Layers
Out Size
Stride
Input: z
1xlx8
7x7 ConvT(ngf x 8), LReLU
7x7x(ngf 8)
1
4x4
ConvT(nng x 4), LReLU
14x [4x14x(ngf x 4)
2
4x4 ConvT((ngf x 2), LReLU
28x28x(ngf
2
3x3 ConvT(1), Tanh
28x28x1
1
Hyperparameters and training details In the image modeling experiments, for the MLE-LEBM baseline, we LD for T = 60 iterations for prior updates during training with step size of 8 0.4. For test time sampling from LEBMs trained with different objectives, we consistently LD for T' 100 with the step size of 0.4 to draw samples from Po- In the anomaly detection experiments, we follow the same experiment configuration as used in Yu et al. (2024).
We use the linear schedule in and Yu al. (2022) specify of and construct the intermediate distributions {gx k= We the number of stages 3. Specifically, the sequence of {a2)3 is: { = [0.01, 0.69175449,, 0.92238785, 0.99974058] To estimate the Eq. (5), we can use Yo, (2.)
Ar Monte-Carla average: Las ~ 1 Σ log Mitaa (x.) + M Σ log Mr (2,) where Z; 9i+1. Z, qj. We follow Mnih Teh (2012) and set n2 Mnn practice, where M 100 for all experiments; is the batch size. When pre-training the VAE model, we set the weight for reconstruction to [1, 100, 1] on the four datasets and KI. weights to when training jointly with the LEBM, we set the weight for reconstructing x to [5. 1] for the term Egg. [log Ps (x(x)))) on SVHN, CelebA64, CIFARIO 10 and CelebAHQ datasets, respectively. We set the weight for controlling
the KL term between d and Pa. to 1, i.e DKL. 1x..(9011r Ilall (x) Σk-1 DKI. (qi I|l,, ge). Other hyperparameters shared across all the experiments.
27

---

Published as a conference paper at ICLR 2026
Table 10: Network structures of the encoder networks used for the SVHN, CelebA, 10. CelebA-HQ MNIST (from top to bottom) datasets. Conv(ri)Norm indicates a convolutional operation with n output channels followed by the Instance Normalization (Ulyanov et al., 2016). We use nif=64 for all the datasets. LRcLU indicates the Leaky-Rel.U activation function. The slope in Leaky ReLU is set to be 0.2.
Layers
Out Size
Stride
Input: X
32x32x3
3x3 Conv(nif x 1)Norm, LReLU
32x32x(nif x 1)
1
4x4 Conv(nif 2)Norm, LReLU
16x16x(nif x 2)
2
4x4 Conv(nif 4)Norm, LReeUU
8x8x 4)
2
4x4 Conv(nif 8)Norm, LRReUU
4x4x 8)
2
4x4 Conv(nem)JNorm, LReLU
Ixlx(nz)
1
Layers
Size
Stride
Input: x
64x64x3
3x3
Conv(nif x 1)Norm, LReLU
64x64x(nif x 1)
1
4x4 Conv(nif 2)Norm, LReLU
32x32x(nif x 2)
2
4x4 Conv(nif 4)Norm, LReLU
16x1 x 4)
2
4x4
Conv(nif x 8)rrm, LReLU
8x8x(nif x 8)
2
4x4
Conv(nif 8)romm LReLU
4x4x(nif 8)
2
4x4 Nommnnemb)Norm, LReLU
IxIx(nz)
1
Layers
Out Size
Stride
Input: X
32x32x3
3x3
Conv(nif 1)Norm, LReLU
32x32x(nif x 1)
1
4x4
Conv(nif 2)Norm, LReLU
16x1 x 2)
2
4x4
Conv(nif x 4)Norm, LReLU
8x8x(nif 4)
2
4x4
Conv(nif Norm, LReLU
4x4x(nif 8)
2
4x4 Conv(nz)oorn, LReLU
IxIx(nz)
1
Layers
Out Size
Stride
Input: x
256x256x3
3x3 Conv(nif 1)Norm, LReLU
256x256x 1)
4x4 Conv(nif 2)rom,, LReLU
2)
2
4x4 Conv(nif x 4)Norm, LReLU
64x64x(nif x
2
4x4 Conv(nif 4)Norm, LReLU
32x32x(nif x
2
4x4 Conv(nif x 8)rrm, LReLU
16x16x(nif x 8)
2
4x4 Conv(nif 8)Norm, LReLU
8x8x(nif 8)
2
4x4 Conv(nif x N)rmm, LReLU
4x4x(nif 8)
2
4x4 Conv(nz)Norm, LReLU
IxIx(nz)
Layers
Out Size
Stride
Input: x
28x28x3
3x3 Conv(nif 1)Norm, LReLU
28x28x 1)
1
4x4 Conv(nif x 2)Norm, LReLU
14x14x(nif x 2)
2
4x4 Conv(nif 4)Norm, LReLU
7x7x(nif 4)
2
4x4 Conv(nif x 8)rrm, LReLU
3x3x(nif 8)
2
3x3 Conv(nz)Norm, LReLU
1xIx(nz)
1
The parameters of all the networks are initialized with the default pytorch methods (Paszke et al., 2019). We use the Adam optimizer (Kingma Ba, 2014) with 31 = 0.5 and 832 0.999 to train the decoder and encoder networks; the log-ratio estimator network is trained with default (8) 32). The initial learning rates of the generator, encoder and log-ratio networks are 2e-4. We further perform gradient clipping by setting the maximal gradient norm 100 for these networks. We run the experiments on A6000 GPU with the batch size of 128. Training converges within 100K iterations on all the datasets.
28

---

Published as a conference paper at ICLR 2026
C.3 REWARD AND CRITIC LEARNING
We adapt the objectives in prior work as follows:
For DxMI (Yoon et al., 2024), we replace the contrastiveddiereenn--sseee EBM update (Sec. 3.2 of Yoon et al. (2024)) with our objective in Eq. (5), treating the true data distribution as the target and diffusion samples as noise.
(ii) For SiD A (Zhou al., 2024a), we substitute the discriminator loss in Eq. (10) in Zhou et al. (2024a) with our objective in Eq. (5), again using true data as the target and diffusion samples noise.
All experiments retain the original network architectures and hyperparameters unless noted:
(i) DxMI: direct ratio regularization weight 1.5 on CIFAR-10 and 1.55 on ImageNe164x64; on ImageNet64 64, the value function learning rate is 3e-5.
(ii) SiD²A: A: direct ratio regularization weight is 0.5 for CIFAR-10 (conditional and uncondi- tional) and 0.4 for ImageNet64 64; we set 1.2 throughout.
(iii) All remaining hyperparameters (e.g.. sampler learning rates, batch sizes) follow the de- faults, making training iterations directly comparable as a measure of computational cost, particularly for SiD? A.
C.4 OFFLINE BLACK- -LO OPTIMIZATION
Architecture We provide detailed network architecture for the log-ratio estimator Jau, (z, k) in Tab. 11; we adopt the same architecture in Appx. C.2 throughout the experiments. The generator networks for and is a 5-layer MLP structure shown in Tab. 12. We employ single network that directly outputs the (x, 3/) pair to implement I8,x, 93.y- For discrete tasks, the g3.x head outputs logits for reconstruction. The encoder network to embed the input is a 3-layer MLP, as shown in Tab. 12. For discrete tasks, we add an additional embedding layer to map discrete inputs to continuous vectors. The network architectures are shared across different task domains; we only adjust the input size for inputs from different tasks. For function values y and continuous x, we normalize these values for training using the offline dataset maxium Xmax Umax and minimum Xmins Umin' Xiin
Xtrain
Verain
y Manin
Vanin
Hyperparameters and implementation details We use the linear schedule as in Ho et al. (2020) and Yu et al. (2022) to specify of and construct the intermediate distributions {gx k==- We set the number of stages to 6. Specifically, the sequence of {o]) is: {ot] k= = [0.01, 0.3237, 0.5165, 0.6322,007132 0.7734, 0.9997]. We re-weight the objective for each 0g with
Wkk = oo/ II a, to further emphasize stages closer to the final target distribution gK+1, as in Yu et al. (2022). To estimate the Eq. (5), can Monte-Carlo average: Lx (xxx)) z . M+ra (2:) + M E/= log M+roe(2)) (2,) where z; q,+1, qi- We follow Mnih & Teh (2012) and set n2 = Mnnj in practice, where M = 100 for all experiments; n, is the batch size. When training with LEBM, 50 for the term Ege [log (x y v/z)]. We set the weight for KL term between 96 and Pa to 0.75. We apply orthogonal regularization (Brock al., 2016) to the parameters of decoder networks, with weight of 0.001 to facilitate training. These hyperparameters are shared across all the experiments.
1 E log Pay (2.)
nı
nz
set the weight for reconstructing and predicting y to
The parameters of all the networks are initialized with the default pytorch methods (Paszke et al., 2019). We use the Adam optimizer (Kingma & Ba, 2014) with 31 = 0.5 and 83 = 0.999 to train the generator and encoder networks; the log-ratio estimator network is trained with default (31, 82). The initial learning rates of the generator and encoder networks are 1e-4, and 5e-5 5 for the log-ratio network. We further perform gradient clipping by setting the maximal gradient norm as 100 for these networks. We run the experiments on a A6000 GPU with the batch size of 128. Training typically converges within 30K iterations on all the datasets.
When sampling with SVGD for optimization, the number of SVGD iterations T is set to 500. For all experiments, we use RBF kernel k(z, 2q) = exp (- 2バअ ||z 2n (I3) for SVGD, and set the
bandwidth to be h Ilo[(++1)
med" med is the median of the pairwise distance between the current
29

---

Published as a conference paper at ICLR 2026
Table 11: Network architecture for the log-ratio timator fa, (z, k). .N is set to 3 for image modeling and anomaly detection experiments, and 16 for the offline BBO experiments. ReLU denotes the Leaky ReLU activation function. The slope in Leaky ReLU is set to 0.2. For SVHN and CelebA datasets, we use 00. For the CIFAR-I0 10 and CelebA-HQ datasets, use nz-128 and n 22-512, respectively. We use nz-8 for anomaly detection on the MNIST dataset; n 22 is the latent space dimension used for offline BBO experiments.
Table 12: Generator and encoder network archi- tectures. LReLU indicates the Leaky-ReLL activa- tion function. dx is the dimension of input x men- tioned in Tab. 18. For discrete tasks, dx specifies the length of input, and L denotes number of possible discrete tokens.
Generator
Layers
Out Size
Note
Input: z
nz 20
Layers
Output size
Note
Linear, LReLU
128
Indx. Emb.
Linear, LReLU
64
neg slope
stage indx. of N
Linear, LReLU
0.2
Input:
Linear, LRRLU
Sin. emb.
128
Continuous Output
neg_ alo_e 0.2
Linear
dx ++1
output (x, v)
Linear, LReLU
128
Discrete Output
Linear
128
Linear
dx x L++1
num. tokens.
Input Emb.
Encoder
Input: z
1L2
20
neg_slope 0.2
Layers
Out Size
Note
Linear, LReLU
128
Embedding
(for discrete x only)
Linear
128
Input: x
32 x dx
emb, x
fau (z, k)
concat (x, y)
(32x Jdx +
input (x, y)
Input: z, Input& & Indx. Emh.
1, 712 20
Emb, of each input
Linear, LReLU
128
neg_slope
128
Linear, LReLU
128
0.2
Mean head
Concat.
256
Linear
20
neg_slope 0.2
LReLUU, Linear
128
Var. head
Linear
20
LReLJU, Linear Input Skip
N ResBlocks
128
LReLU, Linear
energy score
samples {zi)o=1; this follows the same heuristic as in Liu & Wang (2016) that Σ, k(z,,z,) x n exppp= 242 mod?) 1. We can balance the contribution from its own gradient and the influence from the other points for each sample 2, Of note, the bandwidth changes adaptively across iterations with this heuristic. For stability, we use Adam (Kingma Ba, 2014) with default hyperparemters instead of AdaGrad (Duchi et al., 2011) 1) used in Liu & Wang (2016) to allow for adaptive step size et- The initial step size is within the range of [0.3, 0.5] for different tasks.
30

---

Published as a conference paper at ICLR 2026
D ADDITIONAL EXPERIMENT DETAILS & SUPPLEMENTARY RESULTS
D.1 GAUSSIAN SIMULATION
We validate our analysis in Proposition 3.3 and Theorem 3.2 on a Gaussian location family. Specifi- cally, we consider (x) = [xx; ax La), where E I2 denotes the mean parameter and La is the identity covariance. The target distribution is set to g. = N(a",.. (a)", and the noise distribution to qu (0, Ia). Unless otherwise noted, use d = 2 for visualization.
The log-ratio and its gradient admit closed forms:
(x) a 2||oe1? Vala(x) a.
Hence, the noisier NCE gradient is
Va L.((a) Eg
M M+Ta(x)
(x (x-a) Epa
M+Ta(x (x ai).
M
where (x(x)) (x) (x())
As M - oo, this reduces the exact MLE gradient
V LMLE(a) = E, [x] - Ep., [x] = a a.
Setup. We set a- = (1.5, -0.8) and initialize at oxo = (-2.0, Gradients are approximated by Monte Carlo with n 4000 samples per iteration. Optimization performed with step size 0.2 for T 150 iterations. We track several metrics during training, including the distance to the optimum ||a: o I|2 and mean and median of absolute gradient errors. We report results across different noise magnitudes M E {1, 10, 50, 100, 1000), with M = corresponding to vanilla NCE and large M approximating MLE.
Further comparison with Nguyen-Waine Wainwright-Jordan (NWJ) and simple reweighting. We also compare our method with the NWJ objective (Nguyen et al., 2010):
LxwJ ((x) = E.y. [log ro. - E =
Va (a) E., x - ai Enp [xa(x) (x - a:),
as well as a variant of our NºCE objective, where we only use NCCE-style weighting for the negative part of the objective and use the positive part of the vanilla NCCE objective:
rx(x)) (a) = (x) log 1 +a((x)
M
(x) log
M + +x((x)
Similar to our previous setting, we consider in this case 5-d gaussians. We set a" 11.1.5.00.76 O., 0.75, 1.5)) and initialize ao =x", i.e., on the opposite side. We use different number Monte Carlo samples to approximate different regimes and for better visual comparison: (i) Gradients are approximated by Monte Carlo with n 500 samples per iteration, corresponding with low gradient estimation variance in Proposition 3.3, and (ii) with n = 2 samples per iteration, corresponding with high estimation variance. Optimization is performed with step size 0.2 for = 150 iterations. We plot again the distance to the optimum ||ax, a *|2 to illustrate the training trajectories. We plot single-run visualization of the optimization trajectory shown in Fig. 5. Further, for more systematic and scientific comparisons, we use the trajectory-wise average squared norm + Σ 0.- 0- as summary of one run. We repeat 100 independent experiments and report the means and stds in Tabs. 13 and 14. We can see that (i) with n 2/500, the optimal- M NºCEs are indeed much better than those NWJs, demonstrated by both lower mean values and lower stds. (ii) When M is sufficiently large, CC empirically approaches the NWJ objective with T logr. r.
Table 13: MSE sweep for Optimal predicted by Prop. 3.3: CV2 Rz 1.414C.
M
NWJ (T lo
1.5
2
5
avg. runs MSE 54.548 + 237.528
0.904 + 0.084
0.884 = 0.083
0.888 + 0.084
0.983 + 0.107
M
10
100
1000
10a
avg. runs MSE
1.139 + 0.168
3.158 2.683
17.564 + 42.751
61.291 + 277.336
31

---

Published as a conference paper at ICLR 2026
PLL Hop www LIK
WCCN Lawoooo
20
140
1.00
1.20
Itoratinn
(a)
(b)
Figure 5: CC gradients approach MLE gradients with appropriate Ms, while NWJ and simple reweighting cannot. We plot trajectories represented by between model and target parameters.
Table 14: MSE sweep for 500. Optimal M predicted by Prop. 3.3: Cv500 = 22.36C.
M
NWJ (T' log r)
10
50
100
avg. runs MSE
1.359 + 5.454
0.678 + 0.004 0.489 = 0.004
0.456 + 0.006
0.453 0.007
1000
10
2 10*
10
avg. runs MSE
0.489 + 0.020
0.641 + 0.324 0.750 = 0.889 1.909 + 11.201
D.2 IMAGE MODELING
For image modeling in Sec, 4.1, we include the following datasets to study our method: SVHN (32 32 3), CIFAR-IO (32 32 3), CelebA (64 x 64 x 3) and CeleAMask-HO (256 256 3). Following Pang et al. (2020a), we use the full training set of SVHN (73,257) and CIFAR-10 10 (50,000), and take 40,000 samples of CelebA as the training data. We take 29,500 samples from the CelebAMask-HO dataset as the training data. The images are scaled to [_ 1] and are randomly horizontally flipped with a prob. of for training.
Table AUPRC(*) scores for unsupervised anomaly detection on MNIST. Baseline numbers are taken from Yoon et al. (2023); Yu et al. (2024). Results of our model are averaged over trials.
Heldout Digit
7
9
AE
0.062 = 0.00
0.204 4: 0.00
0.259 +: 0.01
0.125 1 0.00
0.113 + 0.00
VAE
0.063
0.337
0.325
0.148
0.104
ABP
0.095 = 0.03
0.138
0.04
0.147
+ 0.03
0.138 0.02
0.102 0.03
IGEBM
0.101 = 0.02
0.106
0.02
0.205
0.100
0.079 0.02
MEG
0.281 = 0.04
0.401 0.06
0.402
0.20
0.003
BiGAN-o
0.287 = 0.02
0.443 + 0.03
0.514
0.04
0.307 100700000
ABP-LEBM
0.336 = 0.01
0.630 + 0.02
0.61
0.413 0.00
JVAEBM
0.297 0.03
0.723 0.04
0.00
0.383 0.00
Adaptive
0,531 = 0.02
0.729
= 0.02
0.742
0.620
0.499 1.9900011
NAF
0.802 0.08
0.648
0.05 0.05
0.00
0.789+0.00
0.441 0.07
MPDR-S
0.764 + 0.05
0.823
0.02
0.741 0004
0.857 0.02
0.478 + 0.05
MPDR-R
0.844 + 0.03
0.711 + 0.03
0.757 0.02
0.850 1 0.01
0.569 1 0.04
DAMC
0.684 = 0.02
0.911
0.01
0.939 + 0.02
0.801 + 0.01
0.705 + 0.01
DAMC-NCE
0.702 = 0.01
0.829 + 0.02
0.764 + 0.01
0.605 + 0.01
0.502 + 0.02
DA M- 1=100, K=1
0.910 - 0.02
0.911 0.01
0.935 + 0.01
0.779 + 0.01
0.699 + 0.02
M=100, K=3
0.959 + 0.01
0.935 + 0.01
0.959 + 0.02
0.845 + 0.01
0.854 + 0.01
D.3 ANOMALY DETECTION
Experiment settings For anomaly detection in Sec. 4.1, we consider the MNIST (28 x 28 x 1) dataset; follow the experimental setting in Zenati et al. (2018); Yu et al. (2024) and use 80% of the
32

---

Published as a conference paper at ICLR 2026
(a)
(b)
Figure 6: Uncurated 1-step generation results from our distilled EDM models. We present 1-step generation results from our distilled models on CIFAR-10 (left) and ImageNet64x64 (right) datasets.
in-domain data to train the model. We base our method upon the DAMC (Yu et al., 2024) model for posterior inference, where the LEBM learned by NºCE as plug-in replacement of the original prior model. With properly learned models, the posterior Po. o (z)x) could form a discriminative latent space that has separated probability densities for normal and anomalous data. Given the testing sample x, we use log of unnormalized joint density Po. [z)s x) x pe. (x, z) = P Z) (z) (2)lx=gs( (x/x) as our decision function; we draw samples from q (z) x) and compare the corresponding reconstruction errors and energy scores. A higher value of log joint density indicates higher probability of the test sample being a normal sample.
method against three groups of strong counterparts: data-space energy-based models, including IGEBM (Du & Mordatch, 2019), MEG (Kumar et al., 2019), IVAEBM (Han et al., 2020) and MPDR-S/R (Yoon al., 2023); ii) latent variable models, including AE, VAE (Kingma & Welling, 2013), ABP (Han et al., 2017), ABP-LEBM (Pang et al., 2020a), Adaptive CE (Xiao & Han. 2022), NAE (Yoon et al., 2021) and DAMC (Yu et al., 2024), and iii) GAN-Based method like BiGAN-a (Zenati et al., 2018)
Baselines We benchmark
D.4 REWARD AND CRITIC LEARNING FOR DIFFUSION DISTIILLATION
Table 16: Ablative FID scores M on CIEAR-10 under Yoon al. (2024) setting.
M
50
100
A 200
3.93
3.05
2.99
3.04
Table 17: Ablative uncond. FID scores of M on CIFAR-10 under Zhou al. (2024a) setting.
M
M
10
M
50
M 100
1.53
1.51
1.45
1.50
We further provide ablation of M on CIFAR-10 in Tabs. 16 and 17. We can see that greater M, e.g., M 50 M 100 is significantly better than smaller M. especially M 1. However, larger M may also incur larger approximation variance based on our analysis in Proposition 3.3. We therefore choose M 100 and M 50 respectively after the sweep for these experiments. These results further confirm our assumption about the noise magnitude.
33

---

Published as a conference paper at ICLR 2026
D.5 OFFLINE BLACK-BOX -Bx OPTIMIZATION
D.5.1 PROBLEM STATEMENT
Let h : X - R be a scalar function; the domain X' is an arbitrary subset of Rd In Black-Boo Box Optimization (BBO), is an unknown black-box function; we are interested in finding that maximizes h:
x" E arg inag l(()).
(23)
XEX
Typically, is expensive to evaluate. In offline BBO, we assume no direct access to during training, and are thus not allowed to actively query the black-box function during optimization, unlike in online BBO where most approaches would employ iterative online solving schemes (Kong et al., 2023b;a; Shahriari et al., 2015; Snoek al., 2012). Specifically, the offline BBO setting (Trabucco et al., 2022), one has only access to a pre-collected dataset of observations, D = {(x,, Mr) ) i== where h((;;) y/- During evaluation, one may query the function h for a small budget of Q queries to output candidates with the best function value obtained.
D.5.2 LEARNING LATENT VARIABLE MODELS FOR OFFLINE BLACK-BAL -Box OPTIMIZATION
x3. (x(z)
Tak
9K
Tag
Q1 Qa
qK
g ( 2z x. y)
ZK
Z1
Zo
0ck l = O.
K
y
Pa(z) qu (z) I Fas (z)
Pg.y (ylz)
Figure 7: Graphical illustration of our BBO framework. construct an energy- latent space model pa for oflline BBO via learning a series of ratio estimators {Fa, J with the NºCE objective Car-sso se to optimize the ELBO withow MCMC. After training. we employ stochastic samplers like LD or SVGD to perform BBO by sampling from the implicit inverse model po (xly) x Eyo(alu) (x/z(( where po (zly) x ( (z) given y. Best viewed in color.
We consider again variational learning of the latent variable model. Specifically, the ELBO this setting would be
ELBOg = log (x v) - DEL (1g ()lo v))
(24)
(2)x.)) [lgg p.3 y|2)] (1 (z|x, u) (z))..
where in (x y). (xxx = ((gax(z)), (2) for continuous x and multinomial for discrete x, and (y/z( = (g I)). g3.x and gs.y is the generator network for x and prediction network for respectively. o.. of take pre-specified values. (z/x, y) = Nus IœI) is the amortized posterior. See Fig. for the graphical illustration.
Optimization as gradient- based sampling Once the latent variable model with Pa is prop- erly trained with NºCE, we can sample from the parameterized inverse model po (xly) x Epe(aly) (x/s)(((zz)) to solve the BBO problem. Specifically, we i) sample z from po xc Ps.y (y/z) (z) with LD or SVGD, where is set to the offline dataset maximum Vmax as in (Krishnamoorthy et al., 2023); and ii) feed them into the generator network 93.x obtain the proposed input designs x. Intuitively, (zly ymmax) exploits the offline high value input design modes, while sampling from po and mapping from z to x with p/s,x (99xx((1x (x|z) achieves expanded latent exploration guided by the latent space model Pa. We provide an interpretation of latent exploration in Appx. D.5.3.
Stein Variational Gradient Descent (SVGD) A well-established counterpart of MCMC for ap- proximating the target distribution is SVGD, first proposed in the seminal work of Liu & Wang (2016). Specifically, it tackles the sampling problem with set of particles for approximation, of (functional) gradient descent is performed to minimize the Kullbeck-Leible Divergence (KLD) and drive the particles to fit the true target distribution. Given unnormalized log-density function
which form
34

---

Published as a conference paper at ICLR 2026
((z), and an initial set of samples {z]]²,, Ji=1+ we can iteratively update these samples as follows to approximate the distribution specified by l:
z: 2t zi + (z))
O,I, where
(z(z) - Σ k(z) z) 2))z) + Vay (z)) 2)
(25)
where -). is a positive definite kernel (e.g., RBF kernel) and & the step size at the I-th iteration. We formulate the optimization process as sampling from an unnormalized log-density and utilize the mode-exploration ability of SVGD to propose high-quality candidates for the BBO problem.
D.5.3 INTERPRETATION OF OPTIMIZATION AS GRADIENT- BASED SAMPLING
In order to extrapolate as far as possible, while still staying on the actual manifold of high-value observations, we need to measure the validity of the generated samples as in Trabucco et al. (2021). We identify the value of y from (,,y(ylzz) (forward model) serves as one good indicator as in Kumar Levine (2020). The generated samples associated with function values similar to existing y values in the offline dataset, are likely in-distribution; those where the ps.y (y) (z) predicts a very different score (often erroneously large, see Fig. when y 1) can be too far outside the training distribution (Kumar& & Levine, 2020; Mashkaria et al., 2023; Krishnamoorthy et al., 2023). In addition, ideally, after training we have a well-learned and informative prior model Pa that i) captures the multiple possible modes of the one-to-many inverse mapping Po (xly), and ii) faithfully assigns the function values observations through the joint latent space (Fig. and 4). Taking together, we can optimize over in the latent space to find the best, most trustworthy x subject to the following constraints: i) has a high likelihood under the prior and ii) the predicted function value associated with z not too different from the value Vmax This can be formulated as the following optimization problem:
arg max y gz(z) s.t. Pa (ymaxx (2) E1, Pa (z E2.
(26)
v(*)
The above constrained optimization problem is conceptually similar to the formulation mentioned in Kumar Levine (2020) Sec. 3.2. The p > é1 constraint enforces that stays on the manifold of high-value observations as we discussed; the (z) > 62 constraint encourages extrapolating beyond the best score the offline dataset D by providing meaningful gradient during the optimization.
In our set-up, the unnormalized log-density of p3.. (1/z))(2 given the best offline function value Vmax can be written as - Alllmaxx 93., + A2 Ta (z) 1 _11211) where X₁ is the variance of (y.((ylz) = (gs,,, (z), I) and A2 re-weights the prior term. This is equivalent to optimizing the Lagrangian of Eq. (26), with the density constraints modified to be log-densities. In practice, we employ LD or SVGD to fully utilize the informative latent space during optimization. Ideally, the dual variables, År and X2 are supposed to be optimized together with y and z, however, we find it convenient to choose them to be fixed as a constant throughout, at A 20.0 and X2 = 1.0, analogous to penalty methods in constrained optimization. The same values of X and X2 are used for all domains in our experiments.
D.6 2D BRANIN FUNCTION
Branin is a well-known function for benchmarking optimization methods. We consider the negative of the standard 2D Branin function in the range .1 E [-5, 10), x2 E [0, 15]:
(2),
-a(x2 bu? + c - r) s(1 cos
(27)
where a 1, b 5.1 global maximas, (-7, 12.275). (R, 2.275). and (9.425, 2.475); with the maximum value of -0.398 (Fig. 2). The Gradient Ascent (GA) baseline uses the offline dataset to train a forward prediction model parameterized by 2-layer MLP mapping x to y and then performs gradient ascent on x to infer its optima. DDOM and BONET follow their default implementation. For the typical forward method, GA in the data space, the problem of assigning clear even in this 2D toy example; the proposed points often violates the square domain constraints. This can be partly seen in the largest std. in Tab. 6.
2. c 5 =- 6= 10, and & this square region, for has three
input with erroneously large value is
35

---

Published as a conference paper at ICLR 2026
D.7 DESIGN-BENCE
D.7.1 TASK OVERVIEW
The goal of TF-Bind-8 and TF-Bind-10 is to optimize for DNA sequence that has maximum affinity to bind with a particular transcription factor. The sequences are of length for TF-Bind-8 and 10 for TF-Bind-10, where each element in the sequence is one of 4 bases. ChEMBL optimizes drugs for specific chemical properties. D' D'Kitty and Ant Morphology. one need to optimize for the morphology of robots. In Supereonductor, the goal is to find material with high critical temperature. These tasks contain neither personally identifiable nor offensive information. We present detailed information the tasks evaluate on in Design-Benche (Trabucco et al., 2022) below in Tab. 18. SIZE denotes the training dataset size, and NUM. TOKS denotes the number tokens for discrete tasks.
Table 18: Design-Bench dataset statisties.
TASK
SIZE
INPUT DIM.
NUM. TOKS
TASK MAX
TF-Bind-8
32898
8
4
1.0
TF-Bind-10
10000
10
4
2.128
ChEMBL
441
31
185
443000.0
D'Kitty
10004
56
340.0
Ant 10004
60
590.0
Supervonductor
17014
86
185.0
D.7.2 HOPPERCONTROLLER & NAS
As in Krishnamoorthy al. (2023) and Mashkaria al. (2023), we exclude the HopperController task in our results due to significant inconsistencies between the offline dataset values and the values obtained when running the oracle (see Fig. and 9). This is a known bug in Design-bench (see details in the link). Therefore, we decided not to include Hopper in our evaluation and analysis. Following Krishnamoorthy al. (2023), we exclude NAS due to excessive computation cost required beyond our budget for evaluating across multiple seeds.
1400
Dataset Values
Unnomalized Function Value
300
Oracle Mean Value
1200
250
Frequency
200
1000
800
100
50
0.0
0.2
0.4
0.5
0.3
1.
2
1d
Normalized Function Values Figure 8: Histogram of normalized function values in the Hopper dataset. The distribution highly skewed towards low function values (plot from Mashkaria et al. (2023)).
Figure 9: Dataset values vs Oracle values for top 10 points. We from mean and standard deviation over 20 runs that oracle is noisy (plot from Mashkaria et al. (2023)).
D.7.3 BASELINES
As mentioned in Sec. 4.3.2, we compare our method with three groups of baselines: i) sampling via generative inverse models with different parameterizations, including CbAS (Brookes al., 2019), CbAS (Fannjiang & Listgarten, 2020). MIN (Kumar & Levine, 2020) and DDOM to validate our model design; ii) gradient(-like) updating from existing designs, egg., GA-based (Fu
36

---

Published as a conference paper at ICLR 2026
& Levine, 2020; Trabucco et al., 2021: Chen et al., 2022; Qi et al., 2022; Yuan et al., 2024; Chen et al., 2024) and BONET and iii) baselines mentioned in (Trabucco et al., 2022). The generative inverse models learn po (x(x)) with different parameterizations and then perform conditional sampling of the high-value designs for BBO. CbAS (Brookes et al., 2019) directly models (o(l)y), while introducing the joint distribution of x and z to facilitate importance sampling. It trains VAE model to parameterize the distribution using identified high-quality inputs x (by thresholding); it gradually adapts the distribution to the high-value part by refining the threshold. MIN (Kumar & Levine, 2020) learns po (x (y) x Ep(aly) [p(x y)1 with a conditional GAN-like model to instantiate p(x|z, 2, y) (Goodfellow et al., 2020; Mirza & Osindero, 2014). The method optimizes over z given the offline dataset maximum ymax and map back to for BBO solutions. The optimization process over z draw approximate samples from p(zly). DDOM (Krishnamoorthy et al., 2023) consider directly parameterizing the inverse mapping pe (xly) with conditional diffusion model in the input design space utilizing the expressiveness of DDPMs (Ho et al., 2020). The second group includes the gradient(-like updating methods, and has the flavor of forward methods in general. the learned surrogates to propose new input designs via simple gradient ascent; (ii) COMs (Trabucco et al., 2021): regularizes the NN- parameterized surrogate by assigning lower function values to designs obtained during the gradient ascent process; (iii) ROMA (Yu et al., 2021b): incorporates prior knowledge about function smoothness into the surrogate and optimizes the design against the proxy; (iv) NEMO (Fu & Levine, 2020): leverages normalized maximum likelihood to constrain the distance between the surrogate and the ground-truth; (v) BDI (Chen al., 2022): proposes to distill the information from the static dataset into the high-scoring design; (vi) 1OM (Qi et al., 2022): enforces the invariance between the representations of the static dataset and generated designs to achieve a natural trade-off; (vii) ICT (Yuan et al., 2024) explores using pseudo-labeler to generate valuable data for fine-tuning the surrogate. BONET (Mashkaria et al., 2023) trains a transformer-bes based model to mimic the optimization trajectories of black-box optimizers, and rolls out evaluation trajectories from the trained model for optimization.
The baseline methods include: (i) Gradient Ascent: operates directly
In addition to these recent works, we also compare with several baseline methods described in Trabucco et al. (2022): 1) GP-qEI (Wilson et al., 2017): instantiates BO using a Gaussian Process as the uncertainty quantifier and the quasi-Expected Improvement (q-El) acquisition function, 2) CMA-ES (Hansen, 2006); an evolutionary algorithm that maintains a belief distribution over the optimal candidates; it gradually refines the distribution by adapting the covariance matrix towards optimal candidates using feedback from the learned surrogate, and 3) REINFORCE (Williams, 1992): first learns proxy function and then optimizes the input space distribution by leveraging the proxy and the policy-gradics estimator. Since we are in the offline setting, for active methods like BO, we follow the procedure of Trabucco et al. (2022) and optimize on surrogate model trained on the offline dataset.
For Q 128 results in Tab. 23, we additionally include ExPT (Nguyen et al., 2024) and BOOTGEN (Kim et al., 2024) for rough reference. Of note, these two methods have slightly different set-ups, and are therefore not directly comparable other baselines and our method. ExPT (Nguyen et al., 2024) focuses extensively on few-shot learning scenarios with models pre-traines on larger datasets. BOOTGEN (Kim et al., 2024) focuses specifcially on optimizing biological sequences.
D.7.4 ADDITIONAL RESULTS & ANALYSIS
Proof-of-concep results for data efficiency in BBO First, we uniformly sample N 5000 and N = 50 points from the branin function domain for training. We observe that our method i) performs consistently better than BONET (Mashkaria et al., 2023) and gradient ascent, and ii) demonstrates less performance drop compared with BONET when dataset size is significantly reduced. Further, we use the D' Kitty dataset and withhold an x% size subsection of data, i.e 0% represents the full dataset. We have similar observation on these datasets summarized in Tabs. 19 and 20, where our method shows much less performance drop compared with the transformer- based method BONET.
D.7.5 ADDITIONAL RESULTS FOR Q = 256
Unnormalized results For reference, we present the unnormalized results of Tab. 7 in Tab. 21.
37

---

Published as a conference paper at ICLR 2026
Table 20: Results on (reduced size) D D'Kitty dataset. x% indicates the proportion training data withheld, i.e. 0% represents the full dataset.
Table 19: Results on (reduced size) Branin fune- tion dataset.
(hest)
Grad. Ascent
BONET
Ouns
5000
-6.199
-3.95 +: 1.26
-1.79 4 0.84
-0.41 1
6.231
-.64 + 3.17
-2.13 0.15
0.43
BONET
285.11 1 15.13
274 1.57
241.17 18.07
Ouns
293.33 1 7.45
280 67 1. 7.86 283 65 9.23
Table 21: Results on design-bench. Unnormalized results with a budget Q = 256.
BASELINE
TFDINDS
TYBINDIO
CHEMBL
SUPERCON.
ANT
D'KIITY
1 (bes))
0:459
0.00532
383700.000
74.0
165.326
190.363
GP-qEl
0.82
1LD86
0,675 d: 0.041
387350.000
11,000
92,696 3.944
JHILI49 d. 0.008
213.816 0,000
CMA-ES
0.953
0.055
0,818 + 0.136
358400 060
$00,000
80-821 +-0666
1010.409
906 906.407 407
4,700 + 2.230
REINFORCE
0.059
0.015
0,072 + 0.313
358400.000 000
21000000
89.027
3.068
-151,907 41.005
-301.306 264284
Gradient Ascent
0
0.015
0.770
0 154
300050 000 2000.000
03,252 5.6
-54,955 33. 482
226.
21,120
COMs
0.120
0.750 0.07%
1901200.000 000 600,000
78.178 6.179
540.623 20.205
7.790
391000.000
653
ChAS
0.88 01118
0.761
B8V000
G000000
83.178 15.
468.711
14.203
MINS
0.88
BL1147
0.770 0.177
89,4000 8.227
17.4888
272:675
DDOM
0:871
0.005
0.885 0.907
3870050 000 10500000
600
11.725
250-599
Ours
0.090 1 0.0003
1:344 L 0.330
392149 225 99211853990
104.848 3:113
372,490 7.251
293.327 7.452
50-th percentile results We present the normalized results with Q 255 at 50-th percentile to further demonstrate the effectiveness of our approach in Tab. 22.
Table 22: Normalized results on design- 50-th percentile results with Q = 256. Baseline numbers from
Mashkaria et al. (2023). DDOM (Krishnamoorthy et al.. 2023) does not report 50-th percentile results; we are unable to find public model weights corresponding with the results in Tab. for DDOM and therefore omit DDOM in this table.
BASELINE
TEBINDS
TFBIND10
D'KITTY
SCORE"
MNR
GP-qFI
0.443
0.491
0.230 0.972
0.272 0.000
0.764 0.004
0,633 +0.000
0-481 0.016
CMA-ES
0,683 0.011
0.376 1 0.001
-0.051 0.004
0.685 0.018
0.633 0.000
0.488 0.020
5.2
REINFORCE
1.0.03
0.472 1: 0.000
0.470 0.017
0.146 1. 0.009
0.307
1 0.002
0.633
0.000
0.083
0.004
5.7
Gradient Ascent
0,572 1: 1.0.024
0.470 1. 0.031
0.865 1. 0.022
0.141 1.0010
0.637 1 0.148
0.633 1 0.000
0.478
0.020
5.1
COMs
0.452 0,003
11472 4: 0,012
0 :: 0.1225
IL5225 2: 0.018
0,885
2 0.002
0.6 0.000
0,522
0.334
4.6
BONET
0.505
0.053
(1696 1: 0.037
0,389 1: 0.015
0.819 =: 0.032
0.907 1 0.020
0.620 2: 0.000
0,614 1: 0,005
3.4
CbAS
0.422 0.007
0.418
0,384 0.010
0.003
0,633 0,000
0.426 0,008
MINS
0.425
0.011
(1.711 0.004
0,330 1: 0.011
0.651 0.010
0.800 0.003
0.633 0.000
0.547 0,005
Dus
0.806
0.507
0.378
0.715
0,907 0.008
D.7.6 ADDITIONAL. RESULTS FOR Q = -128
25-th, 50-th, 75-th and 100-th percentile results We present the normalized results with Q = 128 100-th and 50-th percentile to further demonstrate the effectiveness of our approach in Tabs. 23 and 25. Additionally, we provide results at 25-th and 75-th percentiles for reference in Tab. 24.
D.7.7 EXTENDED DISCUSSION FOR ABLATION STUDY
As mentioned in Sec. 4.3, we provide ablative results on noise magnitude M, number of intermediate stages K and query budget Q Tabs. 26 and 27, Of note, the K M 100 row in Tab. 8 means learning a LEBM w/ Eq. (5) alone, w/o ratio decomposition. We can see that the variant delivers significantly better results than MLE-LEBM w/ SVGD sampler. This indicates the effectiveness of the noise- intteeintensified objective. K = 6, M = 1 row means learning with the original NCE objective but with ratio decomposition. w/o intensifying the noise distributions. We can see that simply combining the NCE objective with variational learning of LEBM only achieves inferior performance compared with learning LEBB with the NºCE objective.
We further provide detailed ablation of M in Tab. 26. We can see that greater M, egg., M = 100 or M = 1000 is significantly better than smaller M, e.g., M = 1 or M = 10. However, larger M incurs larger memory consumption based on our implementation in Appx. C. It also has a pitfall of leading to larger approximation variance based on our analysis in Proposition 3.3. We therefore choose M = 100 for all experiments. These results further confirm magnitude. We can see from Tab. 26 that larger number of intermediate stages K typically brings better performance. These results demonstrate the need for ratio decomposition, aligned with our
assumption about the noise
38

---

Published as a conference paper at ICLR 2026
Table 23: Normalized design-bench results with Q = 128 at 100-th percentile. Baseline numbers from Kim al. (2024); Yuan et al. (2024); Nguyen et al. (2024); Chen al. (2024). ChEMBL dataset is excluded following baseline set-ups. iiniccttes that the baseline has slightly different set-ups; the numbers are for rough reference (sec Appx. D.7.3 for more dctails).
BASELINE
TFBINDS
TFBINDI0
SUPERCON.
ANT
D'KITTY
MEAN SCORE'
MNR
D (bess)
0.439
0.467
0.399
0.565
0,881
GP-GEI
0.802 0.024
0.596 0.000
14.2
CMA-ES
0.853 1: 0,022
0.670 1: 0.023
0. 0.965 0.024
1:214 0.732
0.721 0.001
0.805
8.6
REINFORCE
0.948 + 0.028
0.663 + 0.034
0.481
0.013
0.266 + 0.032
0.562 + 0.196
0.584
12:4
"BOOTGEN
0.979 + 0.001
"ExPT
0.036
0.677 0.048
0.970 0.004
0.973 0.005
Grad. Aspent
8.886 + 0.035
0.647
0.495
00011
0.9134 0.0011
0.944 0.017
0.781
10,2
COMS
0.496 + 0.065
0.622 + 0.003
0.491 + 0.028
0.856 + 0.040
0 = 0.015
0.681
13.4
BONET
0.911 + 0.005
0.756 +: 0.006
0,500
0.002
0.927 =: 0.002
0.954 = 0.000
0.8.1
6.8
ROMA
0.921 1 0.040
0.006 0.035
0.510 0.015
0.917 0.030
0.927 0.013
0,789
8.2
NEMO
0.502
0.002
0.958 0.011
0.007
BDI
0.870 :: 0.000
0.605 +: 0.000
0.513 0.000
0.906 :: 0.000
0.919 0.000
1.763
11.6
IOM
0.878 + 0.060
0.648 + 0.023
0.520 + 0.018
0.918 + 0.031
0.945 + 0.012
0.782
8.6
ICT
0.958 + 0.008
0.601
+ 0.023
0.503 # 0.017
0.961 + 0.007
0.968 = 0.020
0.816
3.4
Tri-mtring
0.970 0.001
0.722
0.514 0.018
0.948 0.014
0.906 0.010
0.824
CbAS
+
0.503 0.069
0.876 0.031
0.008
0.770
10.8
Auto.CbAS
0.910 1: 0.044
0.630 J: 0.045
0,421 0.045
0.882 0.045
0.906 2: 0.006
0.745
13
MINS
0,905 1. 0.052
0.616 1: 0.021
0.499 0.017
0.445 1 0,080
0.892 1: 0.011
0.671
13.0
DDOM
0.957 1 0.006
0.657
0.006
0. 0.405 0.012
0.940 0.000
0.955 4: 0.001
0.797
0.011
0.550
0.954 0.000
0.003
Table 24: Normalized design-bench 25-th and 75-th results with Q 128.
Percentile
TEBINDS
TFBIND10
SUPERCON.
ANT
D'KITTY
25-th
0.635
0.020
0.453
0.007
0.351
0.019
0.527 0.033
0.878 0.002
75-th
0,873
0.006
0.945 + 0.012
0.409 = 0.013
0.832 + 0.012
0.910 1 0,002
Table 25: Normalized design-bench 50-th percentile results with Q = 128 100-th percentile. Baseline numbers from Kim et al. (2024); Yuan et al. (2024); Nguyen et al. (2024); Chen et al. (2024). CLEMBL dataset is excluded following baseline set-ups. indicates that the baseline has slightly different set-ups; the numbers are for rough reference (see Appx. D.7.3 for more details).
BASELINE
TFBINDS
TFBINDI0
SUPERCON.
ANT
D'KITTY
MEAN SCORE"
MNR-
D (ibos))
0.439
0.067
0,399
0,565
0,884
GP-EEL
1:000
0.467
0.000
0:015
0.567
0,883
0.531
CMA-ES
0.537
0.014
0.484
0.014
0.370 0.003
-0045-
0.004
0.684 0.016
0.408
10:2
REINFORCE
0.462 + 0.021
0.475 + 0.008
0.463 + 0.016
0.138 + 0.002
0.356 + 0.131
0.379
10.8
"BOOTGEN
0.833 + 0.007
*EXPT
0.473 0.014
0.477+00.00
0.70G 0.018
0.902 0.006
Grad. Ascent
0.532 0.017
0.529
0.027
0 0.015
0.564 + 0.014
0.877
0.005
0.568
7.4
COMS
0.439 0.000
0.366 +: 0.002
0.316
= 0.022
0.508 =: 0.002
0,883
0.002
0.534
10.6
BONET
0.505 0.004
0.465 1: 0.002
0.470 1: 0.004
0.620 :: 0.000
0,857 0.000
0.591
5.4
ROMA
0.555 0.020
0.512 0.020
0.572 1 0.019
0.179 1 0.011
0,853 0.007
0.55d
7.6
NEMO
0.548 0.017
0.516
0.020
0,865
0.539
RDI
0.430 0.000
0.470
0.000
0.412 0.000
0.474 0.000
0.855 0.000
0.546
9.4
10M
0.430 0.000
0.477
+ 0.010
0.352 0.021
0.500 + 0.003
0.876 + 0.006
0.531
9.8
ICT
0.551 0.013
0.541 1: 0.004
0.399
0.002
0.592 1: 0.025
0.871 1: 0.005
0.501
5.2
Tri-mrinn
0.600 0.021
0.527 0.008
0.355 : 0.003
0.606 + 0.007
0,886 0.001
0.507
4.2
CBAS
0.426 0:010
0.463
0.007
0.007
0,753 0.005
0-428
AmB.CCsA
0.410 0.007
0-461
+ 0.0007
0.131 0.010
0.364 + 0.014
0.736 0.0225
0.422
MINS
0.421 + 0.005
0.468 0.006
0.396
= 0.016
0.618 + 0.040
0.887 0.004
0.546
8.8
DDOM
0.553 2: 0.002
0.488
4: 0.001
0.295 0.001
0,500 = 0.003
0 0.001
0.555
8.6
Ours
1. 0.006
0.099
0.002
0.370 0.004
0,704 0.013
0.897 1: 0.003
0.069
analysis in Sec. 3.3. In addition, since we are using the same network architecture for ablating different Ks, the network capacity is fixed. Larger number of intermediate stages may require a larger network to perform well (circumventing competition between stages), as we are learning the ensemble of ratio estimators using the shared network. SVGD in the data space typically requires a relatively large population of initial samples to perform well. In our set-up, we set the query budget Q to the population size and run SVGD in the latent space. We can see from Tab. 27 that our method is robust Q by pulling SVGD sampling back to the latent space.
Random baselines One valid concern is that our framework solves the offline BBO problem by simply memorizing the best points in the offline dataset and proposing new points close to those best points during evaluation utilizing the randomness in the sampling process. To see whether this is the case, follow Mashkaria et al. (2023) to perform a simple experiment on the D' D'Kitty task.
39

---

Published as a conference paper at ICLR 2026
Table 26: Ablation of K, M SPRCON. (avg.
over runs).
Table 27: Ablation of Q D'Kitty (avg. over 5 runs).
M 0.427
0.567
0.5
2
9 Q= 8
Q 32
Q = 128
?
256
K M
M 10
M 100
M 1000
O 1.000
0.920
0.961 0.964.c.444
0:407gnoss
g
0.518goam
We include random baselines by similarly choosing a small hypercube domain around the optimal point in the offline dataset. These baselines then uniformly sample 256 random points in the cube as the proposed candidates; this can be seen as a offline dataset oracle w/ noise model. In Tab. 28, 0.1. width means
we show the results for different widths of this hypercube ranging from returning the best point in the offline dataset. We can see that the best result (226. 10) from these noisy oracle models is significantly lower than the result from our model (304.36). We can also see from the trend in max, mean and std. values that simply extending the width of cube for random search is hopeless for solving the offline BBO task. This is very likely due to the high-dimensionality and eighly-multimddll nature of the black-box function input space. The comparison between these random baselines and our method suggests that our method does find an informative latent space for effectively extrapolating beyond the offline dataset best designs. The prior model in our formulation provides meaningful gradient for exploring the underlying high-function--all manifold.
Table 28: Random baseline results on the D Kitty dataset. Results from 5 runs.
CUBE WIDTH
0.00
0.005
0.01
0.05
0.1
OURS
MAXt 199.23
212.66
222.44
226.10
209.00
304.36
MEANT
199.23
190.68
182.13
- -169.62
-368.71
206.76
STD. 0.00
9.28
12.21
331.00
261.37
63.26
USE OF LARGE LANGUAGE MODELS
During the preparation of this manuscript, we used Large Language Models (LLMs) to assist with language polishing, organization of sections, and improving clarity of exposition. All technical contributions, including theoretical results, experimental design, and analysis, were conceived and carried out by the authors. The LLMS were not used to generate novel research content, proofs, or experimental results.
40


Reviews:
[{"review_id": "NuNjXoWrX0", "reviewer": "Reviewer_Qdxk", "summary": "This paper points out the problem that Noise Contrastive Estimation (NCE), which estimates a density of the data distribution by estimating the ratio between the target density and noise density, underperforms when the ratio is large. This phenomenon commonly occurs when the data is high dimensional or multimodal. The authors propose to rescale the log odds that appear in the NCE objective by constant $M$. When $M > 1$. this corresponds to artificially increasing the magnitude of the noise. This \"Noisier NCE\" or N2CE reduces to the conventional NCE when $M=1$. They show that as $M$ increases the gradient of the N2CE loss with respect to the parameter, approaches to that of the log-likelihood (score function), thus N2CE amounts to maximum likelihood estimation (MLE), both theoretically and empirically. Then it is further shown that this simple remedy to NCE works effectively well in high-dimensional or multimodal settings, and generative modeling without additional cost or structural change of the models.", "questions": "1. In Proposition 3.3, isn't the second term of $V_n$ $\\min\\\\{ E_{q_0}\\\\|\\nabla_{\\alpha} \\log r_{\\alpha}\\\\|^2, M^2 \\\\|\\nabla_{\\alpha} r_{\\alpha}\\\\|^2\\\\}$?\n\n2. Also in Proposition 3.3, the result suggests that there is an optimal, finite, choice of $M$, unless $\\\\|\\nabla_{\\alpha} r_{\\alpha}\\\\|^2 \\gg M^2 \\\\|\\nabla_{\\alpha} \\log r_{\\alpha}\\\\|^2$.for all $M$. But Proposition 3.1 states that the larger $M$, the better. How can this apparent conflict be resolved?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Theoretical results indicate that that gradient of the population version of the N2CE objective converges to the score function as $M$ increases, under mild regulatory conditions. Under the exponential family assumption, the iteration complexity is bounded by the cube of the condition number of the Fisher information matrix, for $M$ sufficiently large. For a finite $M$, the difference between the score function and the gradient of the empirical N2CE loss exhibits a bias-variance trade-off, where the bias diminishes at a rate $O(1/M^2)$.\n\n2. The proposed method is simple, easy-to-understand, and requires only a modest change in the NCE objective function so there is little additional cost in applying it. Yet it shows an impressive performance boost in image generation, anomaly detection, and offline black-box optimization. For example, in CIFAR-10 and ImageNet64x64, combined with 10-step or 1-step sampling scheme, a SOTA-level result is achieved with half the training iterations.", "weaknesses": "1. The key assumption for the main result that the gradient of the N2CE loss approaches to the score function as $M\\to\\infty$ relies on the assumption that $p_{\\alpha}$ is dominated by $q_0$ uniformly over $\\alpha$. Density chasm is most severe when this assumption is violated, and simply increasing $M$ does not completely resolve this. \n\n2. The choice of the noise density $q_0$ may be much more critical than its scale. Some sort of comparison based on a matrix of $M$ and $q_0$ is warranted. \n\n3. Proposition 3.3 suggest bias-variance trade-off in $M$, but no guide for choosing $M$ is provided. \n\n4. Large-scale benchmarks in limited to visual datasets like CIFAR-10 and ImageNet64x64. It is desirable to test N2CE on the other domains, such s LLM or speech data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6j8kb1isnM", "reviewer": "Reviewer_Qgtk", "summary": "This paper proposes a more general Noise Contrastive Estimation (NCE).\nRecalling the classical sigmoid function $\\sigma(x)$ and the binary-cross entropy is used to compute the NCE loss as:\n$\\sigma(x)=1/(1+e^{-x})$ and\n\n$$\\mathcal{L}_\\text{NCE}(\\theta) = \\mathbb{E}_{x\\sim X}[log(\\sigma(f(x;\\theta)))] + \\mathbb{E}_{\\epsilon}[log(1-\\sigma(f(\\epsilon;\\theta)))]$$\n\nThe method proposed by the paper called Noiser NCE (N$^2$CE), it introduces a new user-provided hyper-parameter $M$ that increases the impact of noise.\nThe sigmoid function and losses are modified as follows:\n$\\sigma_M(x)=M/(M+e^{-x})$ and\n\n$$\\mathcal{L}_{\\text{N}^2\\text{CE}}(\\theta,M) = \\mathbb{E}_{x\\sim X}[log(\\sigma_M(f(x;\\theta)))] + M\\mathbb{E}_{\\epsilon}[log(1-\\sigma_M(f(\\epsilon;\\theta)))]$$\n\nAnd as noted in the paper $\\text{N}^2\\text{CE}(.,M=1)=\\text{NCE}(.)$\n\nThis surprisingly simple modification makes the gradient of the N$^2$CE loss approximate the gradient of Maximum Likelihood Estimation (MLE) when $M\\to +\\infty$.\nIn turns this allows N$^2$CE to converge better and in higher than NCE can as demonstrated by the various experiments.", "questions": "1. I gather that the loss can be seen as a modified sigmoid and a weighted binary cross entropy. But how does inference, do we use a vanilla sigmoid?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The method is simple and easy to implement.\n2. It addresses an important limitation of vanilla NCE: its difficulty to deal with high-dimension data, in which it typically revert to learning the noise directly, not the ratio data-noise.\n3. The connection with MLE is proven.\n4. There's no overhead training cost compared to vanilla NCE.\n3. Experimental results are very convincing, showing large gains over NCE and other methods.", "weaknesses": "1. The paper packs too much information, notably in the experimental settings, and consequently a lot of it has been pushed to the appendix (the paper is 37 pages in total). I tend to prefer paper that focus on one idea and line of experimentation in depth which are then built-on by follow up paper.\n2. Possibly as a consequence of the previous point, background information is severely lacking. For example LEBM and DAMC which are heavily used in experiments are not provided as background, making the paper not self-contained.\n3. As a whole, inference is left out of the paper (apart from saying we use methods A and B for inference, again reinforcing the previous point), only training is covered.\n3. Lack of experimental consistency: sometimes telescoping ratios is used while other time direct ratio regularization is used.\n4. Lack of terminology consistency: telescoping ratios are also referred to as multi-stage ratio and ratio decomposition. This made me doubt my understanding, are they the same thing as I assumed or not?\n5. From the above, I get the feeling it's not one method but two depending on how the ratio is regularized, taking away from the elegance of the idea.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VJaoys4njP", "reviewer": "Reviewer_PhJz", "summary": "The authors explore a simple modification to the Noise Contrastive Estimation (NCE) density ratio estimator: as proposed by Gutmann et al. (2012), a positive factor $M$ is used to reweigh the noise distribution, yielding N$^2$CE. It is claimed that this simple modification results in loss function gradients that get closer to the maximum-likelihood loss gradients as $M$ grows, providing practical improvements over classical maximum-likelihood and standard NCE (e.g., yielding better samplers).", "questions": "1. In the text, you conclude that sufficiently large $M$ (e.g., $50$ or $100$) are necessary for achieving better results: _\"This\nagain supports our analysis that a sufficiently large M is critical for accurate gradient approximation.\"_, line 411. However, with $M$ of this magnitude, N$^2$CE is essentially NWJ. Is there any reason to prefer N$^2$CE with moderate $M$ over NWJ?\n2. Why did you focus on proving the convergence of the gradients, but not of the loss function itself?\n3. Can you compare your method with plain NWJ, DRE-$\\infty$ and Classification Diffusion Models (please, see the weaknesses for the references)?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "The experimental setups are diverse and truly extensive, and the provided results are rich and comprehensive. The writing is also clear and easy to follow.\n\nAdditionally, the authors provide theoretical results, proving their main claim and supplementary convergence results.", "weaknesses": "**Connection with MLE**\n\nThe work is rather weak when it comes to the main claim: the connection between N$^2$CE and MLE, and the novelty behind it. Firstly, this connection is outright obvious from the loss function alone: note that\n$$\n\\log M + \\log \\frac{r}{M + r} = \\log \\frac{r}{1 + r/M} \\underset{M \\to +\\infty}{\\longrightarrow} \\log r\n$$\n$$\nM \\log \\frac{M}{M + r} = -\\log \\left( 1 + \\frac{r}{M} \\right)^M =-\\log \\left( 1 + \\frac{1}{M/r} \\right)^{r \\cdot M/r} \\underset{M \\to +\\infty}{\\longrightarrow} -\\log e^r = -r\n$$\nNow, combining everything and denoting $T = \\log r$, for high $M$ we get\n$$\n\\mathcal{L}\\_M \\sim \\mathbb{E}\\_{q^*} T - \\mathbb{E}\\_{q_0} e^T + \\text{const},\n$$\nwhich is the **well-known** Nguyen-Wainwright-Jordan (NWJ) [1] or KL-$f$-GAN [2] or \"self-normalizing DRE\" [3] variational lower bound on the Kullback-Leibler divergence. Maximizing this bound **is** the maximum-likelihood learning (recall the connection between the KL-divergence and cross-entropy).\n\n[1] X. Nguyen et al, \"Estimating divergence functionals and the likelihood ratio by convex risk minimization\". Proc. of NeurIPS 2008.\n\n[2] S. Nowozin et al, \"f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization\". Proc. of NeurIPS 2016.\n\n[3] B. Poole et al, \"On Variational Bounds of Mutual Information\". Proc. of ICML 2019\n\nNext, the work lacks an important information-theoretic background. Currently, without it, the connection between N$^2$CE and MLE appears as some sort of miracle (*\"As we show below, increasing M has a striking effect...\"*, line 146). However, by showing that N$^2$CE is just a lower bound on some divergence which interpolates between the Jensen-Shannon and Kullback-Leibler divergences, one resolves this mystery.\n\nFrom the theory of variational representations of $f$-divergences (Section 7.13 in [4]),\n$$\n\\text{KL}(p \\Vert q) = 1 + \\sup\\_T [\\mathbb{E}\\_p T - \\mathbb{E}\\_q e^T] = 1 + \\sup\\_r [\\mathbb{E}\\_p \\log r - \\mathbb{E}\\_q r] \\qquad \\text{(MLE, NWJ)}\n$$\n$$\n\\text{JS}(p \\Vert q)  \\overset{\\text{def}}{=} \\frac{1}{2} \\text{KL}(p \\Vert q/2 + p/2) + \\frac{1}{2} \\text{KL}(q \\Vert q/2 + p/2)  =  \\log 2 + \\sup\\_h [\\mathbb{E}\\_p \\log h - \\mathbb{E}\\_q \\log (1 - h)] = \\log 2 + \\sup\\_r \\left[\\mathbb{E}\\_p \\log \\frac{r}{1+r} + \\mathbb{E}\\_q \\log \\frac{1}{1+r}\\right] \\qquad \\text{(NCE)}\n$$\nNow, one can prove that\n$$\nD\\_\\alpha(p \\Vert q) \\overset{\\text{def}}{=} (1 - \\alpha) \\cdot \\text{KL}(p \\Vert \\alpha q + (1 - \\alpha) p) + \\alpha \\cdot \\text{KL}(q \\Vert \\alpha q + (1 - \\alpha) p) = h(\\alpha) + \\sup\\_r \\left[\\mathbb{E}\\_p \\log \\frac{r}{M+r} + M \\mathbb{E}\\_q \\log \\frac{M}{M+r}\\right] \\qquad \\text{(N$^2$CE)},\n$$\nwhere $\\alpha = M / (1 + M)$, $h(x)$ is the binary entropy function, and the optimal $r^*(x) = p(x) / q(x)$.\n\nThat said, of course we get $\\left. \\text{(N$^2$CE)} \\right|\\_{M=1} = \\text{NCE}$ and $\\lim\\_{M \\to +\\infty} \\text{N$^2$CE} = \\text{MLE}$.\n\n[4] Polyanskiy, Y., and Wu, Y. \"Information Theory: From Coding to Learning\". Cambridge University Press, 2024.\n\n**Is there really a need for N$^2$CE?**\n\nNow that we have established that N$^2$CE is just NWJ for sufficiently large $M$, why can't we just use the latter? In the authors' text, I find no justification for using moderate $M$. Therefore, it is logical to jump straight to NWJ, thus avoiding numerical instabilities and recovering the *exact* MLE gradients.\n\n**The \"Density-chasm\"**\n\nWhile the authors address the _density-chasm_ problem from the perspective of the \"magnitude\" of the noise distribution, it is a well-known fact that NWJ (an $M \\to \\infty$ limit of N$^2$CE) still suffers from this exact problem (for instance, please, refer to Figure 2 in [3], that corresponds to the density-chasm case for high mutual information). Therefore, I do not find it convincing that N$^2$CE offers something new to resolve this issue.\n\nFurthermore, while the authors cite the telescoping density ratio estimation technique (TRE by Rhodes et al. (2020)), they do not recognise more recent classifier-based DRE approaches: DRE-$\\infty$ [5] and Classification Diffusion Models [6]. These methods proved to be better at dealing with the density-chasm, while also providing tractable density ratios (and decent generation results in [6]).\n\n[5] K. Choi et al., \"Density Ratio Estimation via Infinitesimal Classification\". Proc. of AISTATS 2022.\n\n[6] S. Yadin et al., \"Classification Diffusion Models: Revitalizing Density Ratio Estimation\". Proc. of NeurIPS 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "czuBQO5ZoX", "reviewer": "Reviewer_LHEq", "summary": "This paper proposes “Noisier” Noise Contrastive Estimation (N2CE), a simple modification of the NCE objective that increases the noise magnitude by a factor $M$. The central claim is that as $M\\to\\infty$, the gradient of the N2CE objective approaches the maximum-likelihood (MLE) gradient. The paper proves a gradient-limit result and a convergence guarantee for exponential families, and analyzes the finite-M, finite-sample error. Empirically, N2CE improves over NCE/MLE-style baselines across a wide range of different tasks.", "questions": "See the Weakness part.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**1. Clear Convergence guarantees.** Proposition 3.1 gives a clean expression showing $\\nabla L_M \\to \\nabla J_{\\text{MLE}}$ as $M\\to\\infty$, and Theorem 3.2 yields polynomial iteration complexity when $M$ is large.  \n\n**2. Simple, drop-in objective.** N2CE just reweights the negative class via $M$; it’s easy to integrate into existing pipelines. \n\n**3. Extensive experiments.** Experiments cover diverse datasets and baselines. It demonstrates versatility across tasks: improving FID scores in generative modeling, boosting AUPRC in anomaly detection, and enhancing diffusion distillation.", "weaknesses": "The method proposed in this work is straightforward: its core idea is to assign greater weight to the loss on negative samples.\n\nFirst, I find the paper’s theoretical results quite confusing. The theory generally requires a fairly large $M$, and in Proposition 3.1, $M$ even tends to infinity. We know that in NCE, the model is expected to give high scores to positive samples and low scores to negative ones. If the loss function assigns an extremely large weight to the negative part, the model is likely to largely ignore the positive part and focus solely on the negatives. In that case, the model could simply assign very low scores to all outputs, which is clearly detrimental to learning. I am therefore not surprised that the experiments indicate $M$ should not be too large. However, the theoretical analysis in the paper suggests that a larger $M$ is preferable. This is not only puzzling but also inconsistent with the experimental results.\n\nSecond, the fundamental idea of the method is to reweight positives and negatives. How is this essentially different from simply multiplying the negative part by a weight $k$? I believe the authors need to discuss the distinction between the two. Do the theoretical results in the paper also hold for such a simple weighted baseline? Empirically, with a straightforward weighting scheme and careful tuning of $k$, can the results match those of state-of-the-art methods?\n\nMoreover, if the choice of $M$ is also important, the paper does not provide a theoretical analysis of how $M$ should be selected. If hyperparameter search over $M$ is required in experiments, the computational cost will inevitably be high.\n\nFinally, since the authors have already conducted image generation of people, why not include generated image results in the appendix, as is commonly done in the experiments of other generative methods?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)