Learning Adaptive Multi-Stage Energy-based Prior for Hierarchical Generative Model
Authors: Jiali Cui
Tian Han
Venue: Accepted by TMLR ()

Abstract: Hierarchical generative models represent data with multiple layers of latent variables organized in a top-down structure. These models typically assume Gaussian priors for multi-layer latent variables, which lack expressivity for the contextual dependencies among latents, resulting in a distribution gap between the prior and the learned posterior. Recent works have explored hierarchical energy-based prior models (EBMs) as a more expressive alternative to bridge this gap. However, most approaches learn only a single EBM, which can be ineffective when the target distribution is highly multi-modal and multi-scale across hierarchical layers of latent variables. In this work, we propose a framework that learns multi-stage hierarchical EBM priors, where a sequence of adaptive stages progressively refines the prior to match the posterior. Our method supports both joint training with the generator and a more efficient two-phase strategy for deeper hierarchies. Experiments across standard benchmarks show that our approach consistently generates higher-quality images and learns richer hierarchical representations.

Published in Transactions on Machine Learning Research (02/2026)
Learning Adaptive Multi-Stage Energy-based Prior for Hier- archical Generative Model
Jiali Cui
jouf/Asteuens.elu
Deportment of Computer Science Stevens Institute of Technology
Tian Han
than6Sstevenssccm edu
Department Computer Science
Stevens Institute of Technology
Reviewed on OpenReview: openreviee. net/ formm?? id= W2zqUkA9Ub
Abstract
Hicrarchical generative models represent data with multiple layers of latent variables orga- nized in a top-down structure. These models typically assume Gaussian priors for multi-layer latent variables, which lack expressivity for the contextual dependencies among latents, re- sulting in distribution gap between the prior and the learned posterior. Recent works have explored hierarchical mergy-based prior models (EBMs) more expressive alternative to bridge this gap. However, most approaches learn only single FBM, which can be ineffec- tive when the target distribution is highly multi-modal and multi-scale across hicrarchical layers of latent variables. In this work, we propose a framework that learns mulli-slage hier- archical EBM priors, where a sequence of adaptive stages progressively refines the prior to match the posterior. Our method supports both joint training with the generator and a two- phase strategy for deeper hierarchies. Experiments across standard benchmarks show that our approach consistently generates higher-quality images and learns effective hierarchical representations.
1 Introduction
Hicrarchical generative models, also known as multi-layer generator models, represent data using multiple layers of latent variables arranged in a lop-down structure. Top layers typically capture high-level semantios, while bottom layers capture Jow-level fine details. Such models have shown strong performance in modelling
complex data distributions and learning multi-level reprosentations (Child 2021; Vahdat & Kautz 2020:
Havtorn et al. 2021 Maaløe et al. 2019). The prior distributions over these latent variables are usually assumed to be Gaussian, which mainly capture inter-layer relationships (i.e., dependencies between layers) and ignore intra-layer contextual relationships (ie.,, dependencies among latent units within the same layer). This lack of expressivity often leads to a mismatch between the assumed prior and the aggregated posterior learned from data, ultimately degrading the quality of generated samples and the capability of learning hicrarchical represcntations.
expressive priors (Cui et al. 2023a b Aneja et al. 2021). EBMs model rich intra-layer dependencies, offering stronger inductive bias and modelling capacity than Gaussians. However, most existing approaches consider only single EBM ("single- stage") for the complex posterior. When the posterior distribution is highly multi-modal and latent scales differently across layers, learning a single EBM prior is difficult and often ineffective. One line of research socks overcome this by borrowing idcas from diffusion models (Cui & Han 2024] Yu et al. 2022 2024). which introduce a sequence of conditional EBMs learned at different noise levels. While effective, these approaches depend carefully designed diffusion schedules and require costly sampling procedures. Another
Recent advances have introduced energy- models (EBMs)

---

Published in Transactions on Machine Learning Research (02/2026)
line of work (Xiao & Han 2022 Rhodes al. 2020 proposes multi-stage learning for marginal EBMs (iiao & Han 2022), which develops a self-adaptive strategy that evolves multiple EBMs without relying on fixed schedule. Yet, these methods are restricted to flat latent spaces, limiting their ability to model hierarchical structures and capture layered representations. As a result, the development of multi-stage hierarchical EBM
priors remains largely unexplored.
In this work, we develop multi-stage learning for hierarchical EBM priors. Our framework constructs a sequence of hierarchical EBM priors, viewed as density-ratio estimators, where each stage adaptively refines the prior by treating the previous stage the new base distribution. This formulation constructs chain of ratio bridges that progressively align the prior with the posterior, yiclding more robust and tractable solution than single-step methods. Unlike diffusion-based approaches, our method does not require a pre- defined schedule, but instead allows both the generator and hierarchical EBMs to evolve self-adaptively. Compared flat latent space Adaptive-CE, the multi-scale nature of hierarchical latent space makes sampling across different layers difficult. To address this, we adopt a uni-scale latent space reparamcterization (Section 3.2 which transforms the multi-scale latent variables into consistent space where EBMs can be learned and sampled more effectively. Our framework supports two training schemes: (i) joint learning, where the generator and EBMs are optimized together, and (ii) two-phase strategy for deeper hicrarchies, where the generator is trained first and the EBMs are learned afterward as complementary priors.
Extensive experiments across benchmark datasets show that our multi-stage hierarchical EBM prior not only improves image quality but also learns semantically meaningful, layered representations. Ablation studies further validate the benefits of our proposed multi-stage learning strategy. Together, these results demonstrate that multi-stage hicrarchical EBMs provide powerful and scalable alternative to Gaussian single-step EBM priors in hierarchical generative modelling.
In summary, our main contributions are as follows:
We propose a novel framework that introduces multi-stage EBM priors for hierarchical generative models, enabling more expressive latent modelling than Gaussian or single-stage EBM prior.
We adopt a uni-scale latent space reparameterizztion and develop two training schemes, which improve the effectiveness and efficiency of EBM learning and sampling.
We perform extensive experiments standard benchmarks, demonstrating that our approach con- sistently generates higher-quality samples and learns effective hicrarchical representations.
2
Preliminary
2.1 Hierarchical Generative Model
Hierarchical generative models extend the standard latent variable generative model by introducing multiple layers of latent variables arranged in a lop-down structure. Each layer captures information at a different level of abstraction, with higher layers encoding global semantics and lower layers modelling fine details. Formally, let X R.D denote an observed data example in a high-dimensional space, and lot z E IR denote latent variables with d < D. A hierarchical model defines a joint distribution over X and a hierarchy of latent variables Z {z).... zz.). For example, this can be done through factorization Sonderby et al. (0016a); Nijkamp et al. (022
(x Z) =P% (X (X T.-1
where
(1)
P350 (Z) = II 1=1
Pa. (z, (z.lzz+1)polzz)
Pa. (X/Z) the generation model parameterized by So, while Po.o (Z) specifies the hierarchical prior over latent variables. Each conditional dependency Ps, (2,(244)) is modcled as a Gaussian distribution (iz; pa, (Z+41), of (Z+41),, where μa, and of denote the mean and diagonal covariance determined by 2et1- At the top of the hierarchy, the prior po(zt) is typically set to standard Gaussian.
2

---

Published in Transactions on Machine Learning Research (02/2026)
Although hierarchical generative models provide powerful framework for learning layered representations through multiple latent variables, their elfectiveness is constrained by Gaussian priors. Such priors primarily capture dependencies across layers in the top-down hierarchy, but they fail to model the richer relationships among latent variables within the same layer, c.g., latent units at the same layer maybe distributed condi- tionally independent. This limited expressiveness leads to a mismatth between the assumed prior and the true aggregated posterior, commonly referred to as the prior-hole problem, resulting in degraded generation quality and learned representations.
2.2 Hierarchical Energy-based Prior
Energy-based models (EBMs) well known for their ability to capture complex dependencies, especially within high-dimensional data or latent spaces (Xie et al. 2016 Nijkamp et al. 2019 Gao
2
EBM Prior
Zi+1
Latent Units
ct al. 2021 Pang et al. 2020a Du ct al.) (021). When applied to
00o
hierarchical generative models, EBMs can act as expressive priors
ZL
Intra-layer Modelling
that model richer intra-layer interactions among latent variables, such as NCP-VAE AAneja et al. (2021) and JointEBM Cui al. (2023a). In particular, JointEBM proposes a hicrarchical EBM prior over all latent variables
Z;
Gaussian Prior
Zi+1
Latent Units
21
1 Zi
g0o Conditional Indeperdent
1
P5, ool (Z)
exp
[Ew (Z)) P
22)
Zw.Bzo
where the energy function (Z) Et- (u, couples the latent variables across all layers, P3 (Z) is the Gaussian prior defined in Eqn and 20,8 is the normalizing constant.
Figure 1: Multi-layer latent variables and latent units.
For learning such hierarchical EBM prior, the learning objective can be viewed as minimizing the KL diver- gence between the aggregated posterior (Z) and the EBM prior pw, (Z):
DKL (cZ)))l...ss where g(Z)
pe
pa (Z Pdata (X)dX
Although this hierarchical EBM prior demonstrates the capability of modelling intra-layer contextual re- lationships (sce Fig providing greater expressivity and capacity than Gaussian priors, learning remains challenging. Specifically, this approach relies on a single-stage EBM prior to bridge the gap between the aggregated posterior (Z) and the base Gaussian prior P8.. PBz.(Z). (Z). However, because (Z) is typically highly multi-modal, forcing single EBM approximate the entire distribution is difficult and often leads suboptimal results.
3 Methodology
3.1 Multi-Stage Density Ratio Estimation
Single-Stage Density Ratio Estimation. Given target distribution g(Z) and base distribution p(Z),
the goal is to estimate their density ratio (Z) ((Z)/p(Z) Gutmann & Hyvärinen (2010). A binary
classifier (Z) : : (0,1) can be trained to discriminate between samples from q (Z) and p (Z) using the standard binary cross-entropy loss:
E Eg(2) (Z)) - Eprz) [log(1 (2))]
(4)
At optimum, (Z) = (Z)+ yielding the estimator (Z) = q(Z)/p(Z) ~ 1-D: We can also maximize an equivalent objective L(w) with respect rw (Z) as
2())
D
E EZ
log
Tw
+ Ep(Z))
1 log
(5)
I T (Z)
1 1 r (Z)
We assume access samples from both ((Z) and p(Z).
3

---

Published in Transactions on Machine Learning Research (02/2026)
This can be viewed as learning an EBM of the form (Z) (Z)()), where Z is the normalizing constant and rw (Z) is an unconstrained positive function. This procedure is widely known noise-cont estimation (NCE). However, when (Z) and p(Z) differ substantially, single-stage NCE often fails to provide
reliable density ratio estimates (Xiao & Han 2022
Multi-Stage Density Ratio Estimation. To overcome the limitations of single-stage estimation, the den- sity ratio can be learned in multiple stages. A sequence of estimators {rw, [2)]==1 constructed, where each stage progressively refines the distribution obtained from the previous one. Formally, the model distribution after stages is defined
71
Putm (Z) = Twm (Z) Puim (Z) = rws (Z) ((Z)
(6)
which can be viewed as a product of intermediate ratio bridges
(Z)
g())
Pum (Z)
Pwi ZZ)
(7)
p(Z)
Pum (Z) Pwm
(())
This decomposition allows for smoother and more tractable learning, especially in the presence of highly multi-modal targets. Prior works have explored this idea in different forms: Rhodes et al. (020 applied a
fixed-stage design in data space, but achieved only limited success on simple datasets (e.g., MNIST). Xiao
& Han 2022) introduced an adaptive multi-stage scheme in flat latent spaces, but such methods remain limited in their capacity to capture structured, hierarchical representations.
3.2 Toward Hierarchical Energy-based Prior
Directly applying multi-stage NCE to hicrarchical latent spaces is non-trivial because of their multi-scale structure and strong inter-layer dependencies. While prior work has studied multi-stage learning in flat spaces (Xiao & Han 2022), extending it to hierarchical FBMs requires addressing these additional challenges. Inspired by the reparameterization strategy introduced in VAEBM (Xiao et al. (021), we adopt their uni- scale transformation that maps the original hierarchical latent space into a uni-scale space W. Unlike VAEBM, which applies this idea to single-stage data-space EBM, we leverage the uni-scale space enable efficient multi-stage learning for hicrarchical EBM priors.
Recall that the Gaussian prior Pisa (Z) in Equ 1 is factorized into consecutive conditional Gaussians, Pa, (z.)zzz)) 2++1) ~ MM (z.+1,, o3, (zi+1)). Through the standard reparameterization trick, each latent vari- able can be expressed as M3. (Zi+1) ) + dg, (Z++1) 1) w, where w; ~ (O,TTa)) Ia.). This defines an invertible mapping Tazo such that Z = Ta (W) and W T (Z). In the W-space, the hierarchical EBM prior can then be expressed as Pa,8so (W) T (T (W))po (see derivation in App.
B.3 where Po (W) is a
standard Gaussian with independent components (w:) ~ (O,,44))
Building on this representation, we define our multi-stage hierarchical EBM prior as
On Z-Space: Pwm. (Z)= II (w, (Z) (0)o (Z) On W-Space: Pum 180 (W) = II rw, (TB.. po (W)
(8)
On Z-Space, the base distribution is Gaussian prior, and the multi-slage modelling aims to approximate the target distribution adaptively and progressively. On W-Space, this formulation leverages the uni-scale roparamctorization to make multi-stage lcarning feasible in hicrarchical EBMs. The transformation T pre- serves hicrarchical dependencies, while the W-space provides a consistent representation that enables better training and sampling. We refer to A.3 for more details.
4

---

Published in Transactions on Machine Learning Research (02/2020)
3.3
Learning Multi- Stage Hierarchical Energy-basee Prior
3.3.1
Joint Learning
Stage
Stage m
Stage 772
Po (z)
Pum 14n-11950 (z)
Pum 30 (z)
new base
base
base
update target
Twn (z)
updato target
w
Twa
p=(z
Pum- 1,0 (l (z)xx)
Pu (z(x)
Figure Updating schema of joint learning. each stage m, a density ratio Tun (Z) is estimated by contrastive learning, distinguishing samples from the prior and from the aggregated posterior. Posterior samples are obtained via MCMC posterior sampling with the prior of the current stage (dashed line), while prior samples can be taken from posterior samples al the previous stage. The estimated ratio is then used to update the prior, which becomes the base distribution for the next stage. By repeating this process, the prior is progressively refined across stages, cnabling stable joint learning of both the generator and the hierarchical EBM prior.
With hierarchical EBM prior, can specify joint distribution of the hierarchical generative model
Pum. X Z) P3. (X(Z)) (Z)
which involves generator parameters 3 and multiple stages of EBM parameters w. Learning can be achieved by iterating between the maximum likelihood estimation (MLE) of 3 and contrastive estimation of w.
Specifically, al the m-th stage, updating the generalor parameters 3 corresponds to maximizing the data likelihood. The gradient takes the form
a 0
Eprata (X)pumm o (2(x)
log P3 X 2) + E (X) (2
log Po (Z)
(10)
03
where Pw... (Z( denotes the generator posterior. These expectations are approximated by posterior sam- pling using Markov Chain Monte Carlo (MCMC), such as Langevin dynamics (sce Equ
14p.
The EBM parameters win are updated via rastivviive estimation, where the generator posterior serves as the target distribution and the prior from the previous stage serves as the base distribution:
Epases(X)pp, (X)p= 1emmmm---eeee (2)x) Zwim
log
Twi
Turn (Z)
+ E
(2)
Owm
log
+ Turm (Z)
(11)
where samples from P ZZ) are also obtained using Langevin dynamics.
Since the hicrarchical latent variables Z are inherently multi-scale, both posterior sampling and EBM sam- pling are carried out the uni-scale W-space. The results are then mapped back Z-space through the transformation function T for gradient computation. This joint learning procedure alternates between up- dating 9 and al each stage, gradually refining both the generalor and the hierarchical EBM prior. More efficiently, once the density ratio estimator achieves sufficient accuracy, the prior samples from Pwm-1, can be replaced by samples from the previous posterior Pum--..f a (Z(X), thereby bypassing MCMC sampling from the prior and reducing computational cost (Xiao & Han 2022).
While joint learning provides a principled way to train both the generator and the hierarchical EBM prior together, it relies hcavily on MCMC posterior sampling. This procodure becomes increasingly incfficient as the hierarchy deepens, since each MCMC step requires backpropagation through all latent layers of the generator.
5

---

Published in Transactions on Machine Learning Research (02/2026)
3.3.2 Two- two-phase Learning for Deep Hierarchy
Stage
Stage m
Stage m + 1
Z
P m
1,500
Pumm (z)
new base
new base
new
base
Twj
Twom
T Twmal
23
(x(z/xx)
Figure 3: Updating schema of two-phase learning. In the second phase, the variational posterior qu (Z(x) serves as the fixed larget distribution for hierarchical EBM prior learning. At each slage m, densily ratio Twn (Z) is estimated between qs (Z and the current base prior. The updated prior Pum 820 (Z) is then used as the base distribution for the next stage.
To address this computational bottleneck, we introduce two-phase learning strategy. In the first phase, we approximate the generator posterior using a variational inference model, which eliminates the need for costly posterior MCMC. In the second phase, our multi-stage EBM prior is learned on top of this fixed variational posterior, retaining the expressiveness of the EBM while ensuring scalability to deep hicrarchical structures.
Variational Learning with Gaussian Prior. Variational learning [Sonderyy al. (2016a) introduces an inference model go (Z) that approximates the generator posterior distribution. For hierarchical latent variables, the inference model is also structured hierarchically, i.e., (Z(X)) go (2) II lo, (2: (), where each conditional factor is modeled as a Gaussian distribution.
First-phase:
With such inference model, variational learning proceeds by maximizing the evidence lower bound (ELBO)
Epanna(X)gs222) log (X Z) Lol (Z
Epani (Xansn())gs [log Pao (X[Z)) Lppaaa. (Z)) (12)
By maximizing the ELBO, effectively maximize lower bound of the true log-likelihood. The gap between the ELBO and the likelihood is given by the KT, divergence term, ELBO = log P (X) (Z) (Z)X), which encourages the variational posterior to approximate the generator poste- rior. In other words, variational learning can be interpreted as approximate MLE, where the inference model provides a tractable surrogate for posterior sampling. Once the inference model is well trained, variational postcrior q (Z serves reliable approximation pa (z) and can therefore be adopted as the target distribution in the second phase of our hierarchical EBM prior learning.
Second-phase: Contrastive Estimation with Fixed Generator. In the second phase, we fix both the generator parameters 8 and the inference network o, and focus on learning the hicrarchical EBM prior. The training objective al stage m is
① log
Two (Z)
Ô log Owm
1
Epiaca (X)s (Z)))
+ Epmm-).e
(13)
iwm + Twm ZZ)
1 + Fulm
ZZ)
where posterior samples are efficiently obtained from the fixed inference model. Conceptually, this phase isolates the task of refining the prior: it learns to bridge the gap between the aggregated posterior qo (Z) and the base Gaussian prior Psso (Z), without requiring costly posterior MCMC updates.
This two-phase design offers two key advantages. First, it makes training more effective in deep hierarchical settings by decoupling generator learning from EBM refinement. Second, unlike most existing methods (Aneja et al. 2021: Xiao al.] 2021 Cui al.] 2023a] that attempt to close this gap with a single-stage EBM, we decompose the process into multiple adaptive stages. Each stage provides refinement of the prior, yielding smoother convergence and more faithful approximation of complex, multi-modal posteriors.

---

Published in Transactions on Machine Learning Research (02/2026)
3.3.3 Markov Chain Monte Carlo Sampling
Both training strategies require access to hierarchical latent variables from either the generator posterior or the EBM prior. To do so, we adopt MCMC sampling, such as Langevin dynamics (Neal et al.) 2011 in the uni-scale space. Specifically, for an arbitrary distribution p(z), it is performed as
a Z++1 2, + dz,
log p(z)) + 225U,
(14)
where I indexes the time step, is the step size and U, is Gaussian noisc.
MCMC for EBM Prior. To sample from the hierarchical EBM prior, we apply Langevin dynamics in I|w-1l" which
a the uni-scale W-space. The gradient torm is computed as aw, Em logra, (T - 2
combines contributions from all ratio functions rw. together with the standard Gaussian reference prior.
MCMC for Posterior. For generator posterior, we also perform Langevin dynamics in uni-scale W-space. & The gradient term is computed as ow;
(W-)) + log Po...... (W+)) (see derivation in App B.3
Variational and MCMC Sampling. MCMC-based posterior sampling requires backpropagation through the generation model, which is often large. When the hierarchy is shallow (e.g., 2 layers), the generator and the EBM prior can be trained jointly with MCMC posterior sampling. In this regime, the cost of gradient propagation remains manageable, and MCMC sampling provides more accurate posterior estimates and better learning signals for the latent variables. However, when the hierarchy becomes deep (e.g., 30 layers), MCMC posterior sampling becomes expensive. Joint optimization of both the hierarchical generator and the hicrarchical EBM prior under these conditions is often unstable. Following prior work Xiao et al. (2021), we adopt a two-phase learning strategy in which a hierarchical variational inference model replaces the MCMC posterior sampler, leading to more tractable and stable training in deep hierarchical settings MCMC sampling in the uni-scale ensures effective sampling for both prior and posterior distributions, while the transformation maps the results back to the original hicrarchical latent space.
4 Related Work
Hierarchical Generative Models. Hierarchical generative models employ multiple layers of latent vari- ables organized in top-down structure and are typically parameterized with Gaussian priors (Vahdat & Kautz 2020 Child 2021 Sonderby et al.) 2016a; Maaløe et al. 2019 Nijkamp et al.) 2020). While con- venient, Gaussian priors lack stalistical expressivity, often leading to the prior-hole problem and degraded generation quality (Aneja et al.) 2021 Cui et al. (203a)). To overcome this limitation, recent studies have explored expressive priors such as EBMs, but most approaches rely on single-stage learning. Our work ex- tends this line of research by developing mulli-stage hierarchical EBM priors that effectively capture complex posterior distributions while remaining scalable to deep hierarchies.
Ensrgy-basod Models. EBMs have been widely applied in data space for modeling imagos and other high- dimensional signals (Xie et al.) 2016 Nijkamp et al. 2019 Du et al. 2021
Gao et al.
2021 Cui & Han 2023
Yu et al. 2024 Schröder bJ. but these methods adopt single-stage estimation, limiting their ability to approximate multi-modal posteriors. Multi- stage training has recently been studied flat latent spaces (Xiao & Han 2022), showing that a sequence of density ratio estimators can improve learning stability. However, extending this idea to hicrarchical latent spaces introduces new challenges due to scale differences and inter-layer dependencies. Our work addresses these challenges with uni-scale reparameterization and develops a self-adaptiv multi-stage framework for hierarchical EBMs, enabling robust and scalable training across deep latent structures.
Xiao et al. 2021p. Extensions to latent space exist in both flat (Pang et al.] 2020a
et al. 2023 Yuan et al.) 2024) and hierarchical settings (Aueja et al.) 2021 Cui et al. 2023a
Diffusion Energy-based Models. Both diffusion frameworks and multi-slage learning approximate complex distributions by constructing a sequence of intermediate models. Diffusion methods learn con- ditional EBMs (c.g., P. (W,[\,,+)) guided by a pre-defined forward noise process (Gao al., 2021: Cui & Han 2024). While effective, this conditioning restricts each EBM local refinements and re- quires carefully designed schedules. In contrast, our multi-stage framework builds marginal models (e.g..
7

---

Published in Transactions on Machine Learning Research (02/2026)
Pum (W) IT== Fe.,
(W)po(W) (W.)ooo in self-adaptive manner, allowing each stage to refine the prior distri- bution without fixed noise schedule. From learning perspective, diffusion-based EBMs are typically trained with MLE, which requires costly inner-loop MCMC sampling at every step. Our approach leverages contrastive estimation, which optimizes density ratios and avoids repeated inner-loop MCMC.
5 Experiments
5.1
Latent Space Modeling
Figure 4: Synthesis on CelebA-HQ-250 (left), LSUN-Church-64 (center), and CIFAR10 (right).
We first evaluate our method on the task of latent space modeling, A well-learned hierarchical EBM prior should produce informative latent samples that, when passed through the generator, yield high-quality and realistic hesis. Concretely, generate images by sampling W from the learned EBM prior, mapping it
into hierarchical latent variables Z = TT (W), and decoding with the generator Pa. Z We benchmark across several datasets, including SVHN Netzer et al. (2011), CIFAR-10 Krizhevsky et al. (2009), CelebA-64
Liu et al. (2014), LSUN-Church-644 Yu et al. (2015). and high-rcsolution CelebA-HQ-2266 Karras et al. (0017)
Image quality measured with Fréchet Inception Distance (FID) and Inception Score (TS).
Table 3: Two-phase learning CIFAR-10.
Table 1: Joint learning for FID (1) comparison.
Methods
(t)
FID
Ours
9,03
7.80
Method
SVHN
CelebA-64
CIFAB-10
Hisraechical Dif EBM Cui & Ham
2024
9.03
6.93
Ours
20.42
29.12
61.12
Joint-EBB Cui
2123
8.99
11.34
Adaptive CE Xino & Han
2022
26.19
35.38
65.01
NCP-VAE Aneja of
24.08
Joint-ERM
26.81
33.00
66.32
NCP-VAE Aurja
D121
33.23
42.07
78.06
Multi-laver Generator
LEBM Pang
NVAE" Vahdat Kantz
2020
5.30
37.73
ED-LEEI Schrmder et
2022
28.10
36.73
73.58
NVAE Recon (TF ID1
0.68
EBIPLA Markss es al 20225
27.59
34.71
79.64
HVAE Senderby et al
2016a
81.44
EVaLP Dutta
35.90
76.43
BIVA Manlle et al.
2019
66.37
Diffusion-A Amor (ied
Ya el
20024
21.17
35.07
60.89
Energy-based Models
2-VAE Wipd
2311
42.07
Dual-MCMC Cui Hau
8.35
9.26
BIVA Musaie db al.
2019
31.65
33.58
66.37
Arclitectural EBM Cuu
21123
63.42
Table Two-phase learning for FID (4) comparison.
DRL.EBM (T
6) Lito el m1
2121
8.40
9.58
Adaptive CR Xian
Flam
65.01
VAEBM Xiso ot al.
A721
8.43
12.19
Methodd
LSUN-Cauuch-6
Hat EBM Hill
19.15
Ours
7.84
6.65
ImprovedCD D'
7.85
25.1
Joint EBA all M1c50
8.76
9,50
9.38
Divergeuce Tringge Hau et ml.
2020
30.10
SCP-VAE Xzojo m
2022
24.70
Adv-EBB Yiu al."
2020
9.10
13.21
NYAE- Vilddtt
Kanty
(203)
20230
30.25
38 13
EVal.P Dutta et
42.30
NVAE UVAAE*Toooo
1.64
2.45
GANs+ Diffusion Models
DRI-EHM
StyleGANv2 w/o ADA Karras et al.
2020
8.99
9.9
GLOW
K
Kingpe K Insrrri 2016
65.03
50.35
Dillaion-Amorlians Amorlized Yu al. 2124
57.72
PGGAN
8.0
6.42
NCSN Song & Ermoon 20019
25.32
ISGG Vahilat 1222
2.10
DDPM
1000
2020
9.46
3.17
Joint Learning. We evaluate our framework under joint training scheme with two-layer (L 2)
hierarchical latent structure, where the generalor and the multi-stage hierarchical EBM prior are learned jointly. We compare against several groups of baselines. (i) Flat latent models with multi-stage EBMs, such as Adaptive-CE Xiao & Han (2022), which learns multi-stage priors but only singlo-layer latent spaces; (it) Hierarchical models with single-stage EBMs, including Joint-EBM Cui et al. (2023a) and NCP-VAL Aneja al. (2021), which extend FBMs to hierarchical structures but rely single-stage formulation;
8

---

Published in Transactions on Machine Learning Research (02/2026)
(iii) Other expressive priors Roussian-based hicrarchical models, such LEBM Pang et al. (020a)). ED-
LEBM Schröder et al. 20023 EBIPLA Marks et. al. (2025). EVaLP Dutta et al. 2025). Hiflusion-Amortias
Yu et al. (2024), 2s- VAE Dai & Wipf 2019), and BIVA Maaloe et al. [2019).
As shown Tab 1 our method consistently achieves superior results across datasets. These results suggest that introducing multi-stage EBM prior into hierarchical latent spaces offers advantages over both single- stage EBMs and flat multi-stage approaches, resulting in stronger priors and improved generation quality,
Two-Phase Learning, For deep hierarchical structures, our multi-stage hierarchical EBM prior is learned as ratio bridges between the fixed aggregate posterior and the base Gaussian prior. Once learned, this EBM prior serves as more informative alternative to the Gaussian prior, enabling generation of realistic samples.
Following the setup of prior works, including Hierarchical Diff-EBM Cui Han 2024), Joint-EBM Cui et al. [2023a], and NCP-VAE Aneja et al. (2021). we build on top of NVAE Vahdat & Kautz (2020) our backbone model (denoted NVAE"). As shown in Tab 2 and Tab 3 our method achieves competitive
superior performance, even when compared to recent diffusion-based approaches. Notably, il yields substantial improvements over the NVAE" baseline, highlighting the efficacy of our multi-stage learning strategy in capturing complex latent distributions.
5.2 Analysis of Hierarchical Representation
A hallmark of hierarchical generalive models is that different latent layers capture different levels of ab- straction: bottom-layer variables tend to encode low-level visual attributes, while top-layer variables capture high-level semantic concepts. In this section, we demonstrate the capability of our multi-stage hierarchical EBM prior in lcarning hierarchical representations.
5.2.1 Hierarchical Controllable Synthesis
Hierarchical Fine-grained Controllability
Top
Bottom
Set
YES
YES Black Hair
Female
Gye-Glasses
Random Sample
NO
Glasses
Black Hair
Figure 5: Starting from random samples, attributes such gender, eyeglasses, and hair color can be nipulated in top-down order. Higher-layer latents govern semantic structure (e.g., gender, identity), while lower-layer latents refine local appearance (e.g., eyeglasses, hair). Importantly, sampling at lower layers preserves the semantics imposed by higher layers, enabling interpretable and fine-grained controllability across the hicrarchy.
An advantage of our framework is the controllability from the multi-stage EBM prior, even when the backbone generator is trained in a fully unsupervised manner. Inspired by (Salimans et al. 2016), which demonstrated how GAN discriminators can be repurposed for classification, we reinterpret each stage of our multi-stage NCE framework as learned discriminator.
Given K class labels or attributes (denoted by v), the hierarchical EBM prior can be extended into multi- class classifier with K + outputs, where the (K + 1)-th logit continues to distinguish posterior samples from prior samples. This formulation enables modcling of the joint distribution ,8n. (W. y) (sce details in App B.4), allowing conditional generation under a specified label y. As illustrated in Fig our hod leverages the discriminative structure of the multi-stage EBM prior to achieve fine-graines and semantically consistent controllability. The hicrarchical organization further ensures that semantic structure imposod by higher layers is preserved when lower layers are sampled, enabling controllable generation that is both structured and coherent across different levels of abstraction.

---

Published in Transactions on Machine Learning Research (02/2026)
5.2.2 Hierarchical Out-of- Dittr--Diittiittionn Detection.
Beyond generation and representation learning, our multi-stage hicrarchical EBM prior also provides a natural mechanism for out-of-distribution (OOD) detection. Each stage of the prior operates as a density- ratio classifier, distinguishing samples from the aggregated posterior versus the base prior in latent space. As a result, OOD detection emerges as a built-in capability of our framework, requiring no additional supervision or auxiliary objectives.
We evaluate this property by training CIFAR-10 the in-distribution dataset and testing on SVHN OOD. Detection is based on the classifier output D Rd + (), where lower values indicate higher likelihood of being OOD. As shown in Table and Figure D AUROC scores increase with depth: top-layer latents provide the strongest OOD signals by encoding distributtionsseeifie semantics, while lower layers capture more generic features that are often shared across datasets Havtorn et al. (2021)
AUROC
120
1,33
L26
L
1-112
AUROC of OOD detection
Stage
0.3312
0.3858
0.4118
0.4312
0.4410
0.9
Stage
Stage 2
Stage 3
0.3501
0.4045
0.4775
0.3953
0.4315
0.4969
0.4626
0.3351
0.5235
0.5713
0.8
Stage 2
Stage 3
AUROC
D>15
L
L
0.6
Stage 1
0,5382
0.6683
0.6735
0.6960
0.7079
Stage 2
0.6416
0.7277
0.7335
0.7471
0.8002
Stage 3
0.6652
0.7835
0.7710
0.8469
0.8897
0.4
Table 4: AUROC (t) for OOD detection. >> k de- notes for using the top & layers. The total number of layers is 30.
130
L33
L-12
L-10
L-21
L-24
L-27
Figure 6: Visualization of AUROC curve.
These findings highlight key advantage of our hicrarchical design. By aligning representation depth with semantic specificity, our model not only improves generative modeling but also enhances robust in detecting OOD samples.
5.3 Analysis of Multi-Stage Learning Dynamic
5.3.1 Multi-Stage Learning Analysis
We next analyze the learning behavior of our multi-stage framework. In our formulation, each stage of density ra- tio estimation refines the prior by building estimator learned in the previous stage, effectively evoly-
Multi Multi-Stage Analysis
the ratio
1.3
ing the base distribution toward the target posterior. As the base distribution becomes progressively better aligned with the posterior, the discrimination task between pos- terior and prior samples naturally becomes harder. This increasing difficulty is reflected higher loss values ob- served in later stages.
1.1
68
23:0
SC00
7300
L0000
12505
15000
12500
20000
Stage
Stage
Stage
Fig 7 illustrates this trend: the average loss (aggregated Figure 7: Visualization of stage-wise evolution across all latent layers) grows as the stage index incrcases. with loss trends averaged across all latent layers. This progression confirms that successive stages refine increasingly aligned priors, making the classification task progressively more challenging. and posterior, while later stages focus on finer adjustments. Such behavior is consistent with theoretical expectations and aligns with prior observations flat latent settings Xiao & Han (002)). These results provide strong empirical support for the effectiveness of our multi-stage learning scheme.
other words, earlier stages correct large mismatches between the prior
10

---

Published in Transactions on Machine Learning Research (02/2026)
Stage
Stage
Stage
1.2
0.6
0.6
122.0
2230
2008
15000
Lx2
Li-22
L100
Lx:30
L-10
L-30
Lo:00
Figure 8: Visualization of layer-wise evolution from stage to stage 3.
5.3.2 Multi-Layer Learning Analysis
To further examine learning dynamics, we analyze loss trends across different latent layers. In hierarchical latent structures, lower layers typically encode fine-grained, low-level details such texture background. These features are oflen more variable and complex, creating larger gap between the aggregate posterior and the base prior. Consequently, the density ratio estimation task at lower layers is relatively easier, yielding lower loss values.
This trend evident in Fig g where lower layers consistently report smaller losses than top layers. As additional learning stages are introduced, the multi-stage EBM prior progressively narrows the gap between the posterior and the base distribution at cach layer. This refinement makes the classification task more difficult across all layers, leading to higher losses over time. The observed progression confirms that our multi-stage framework effectively reduces discrepancies at every level of the hierarchy, supporting consistent prior learning in complex multi-layer latent spaces.
5.4 Analysis of Energy Landscape
5.4.1 Joint Langevin Transition.
We analyze the sampling behavior using Langevin tran- sition. Starting from sample drawn from the standard Gaussian, we perform 10 Langevin steps for learned multi- stage hierarchical EBM priors. As illustrated in Fig 9 im- age quality improves gradually and coherently with each step. This smooth evolution reflects the well-structured energy landscape of our model, which further facilitated by operating in the uni-scale w-space.
Figure 9: Joint Langevin transition.
5.4.2 Progressive Transition
Figure 10: Visualization of progressive Langevin transition.
We next examine whether each stage of the EBM prior can serve as a base distribution for the subsequent stage. To this end, we perform progressive sampling: at the s-th stage, 10-step Langevin dynamics is applied only with the s-th stage EBM prior, initialized from the final sample of the stage. Io shows that each stage produces a refined update of the sample, demonstrating that the sequence of EBMs provides a coherent progression that incrementally shapes the energy landscape.
11

---

Published in Transactions on Machine Learning Research (02/2026)
5.5 Ablation Studies
Multi-Stage vs. Single-Stage under Fixed Iterations. We begin by testing the effectiveness of the proposed multi-stage framework. To ensure fairness, we keep all other settings and fix the total training budget to 15K itcrations. We then compare two scttings: (i) a single-stage hicrarchical EBM prior trained for the full 15K iterations, and (ii) a three-stage model, where 5K iterations are allocated to each stage.
As reported in Tab the multi-stage approach substantially outperforms the single-stage baseline, achieving an FID of 9.48 compared to 15.15. Notably, the table also reveals clear progression: Stage starts weaker due to limited training, but each subsequent stage produces marked improvements, with the final stage delivering the strongest results. This experiment highlights our framework: rather than relying prolonged training of a single-stage EBM prior, decomposing the task into multiple adaptive stages provides a more effective refinement process. Even with the same computational budget, multi-stage learning achieves better generative performance, suggesting that the learning scheme itself is crucial to success.
Table Comparison of single-stage vs. multi-stage learning under a fixed total of 15K training iterations.
Iteration
Single-stage (15K)
Stage (GK)
Stage 2 (GK) Stage 3 (GK)
FID
15.15
18.68
12.87
9.48
Effect of Langevin step. Each stage in our framework employs Langevin dynamics to draw samples from the EBM prior refined by the previous stage. Our default setting uses K 10 Langevin steps, but we also examine the effect of varying K. As shown in Tab performance improves as K increases: reducing to K = 5 yields weaker results (FID 11.15), while incrcasing to K = 15 produces substantial gain (FID 7.11). This trend confirms that more accurate sampling enhances prior quality and improves generative performance. Nevertheless, since larger K values also increase computational overhead, we adopt K = 10 as a balanced default across experiments.
Table Impact of Langevin steps K and number of stages M on generation quality for CelobA-HQ-256.
CIFAR-10
K
5
K
15
K 10,
M
3
M
FID
11.15
7.11
7.84
7.05
Effect of Stage Number. We also analyze the influence of the number of stages M in the multi-stage learning framework. With M = 3, the model achieves strong generative quality (FID 7.84). Increasing the number of stages to M = 4 further reduces the FID to 7.05, indicating that additional stages continue to refine the prior and improve performance. However, the performance gain is relatively modest compared to the increase in training cost and computational complexity. This observation suggests that while deeper staging can yield incremental improvements, three stages provide an effective balance between accuracy and efficiency. Consequently, we adopt M as the default setting for all reported experiments.
6 Conclusion
We proposed a novel framework for learning hierarchical energy-based priors through multi-stage learning scheme. Instead of relying single-stage EBM to approximate the highly complex aggregated posterior, our method progressively refines the prior via sequence of ratio estimators in a uni-scale latent space. Extensive experiments show that our approach improves image generation quality and also demonstrated its potential in hierarchical controllability, out-of-distribbttio detection, and energy landscape analysis.
Acknowledgements
This work is supported in part by NSF IIS- 13-2339604 Any opinions, findings, and conclusions or recommen- dations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. We also thank Zhisheng Xiao for sharing the codebase, which facilitated part of this work.
12

---

Published in Transactions on Machine Learning Research (02/2026)
References
Jyoti Aneja, Alex Schwing, Jan Kautz, and Arash Vahdat. A contrastive learning approach for training variational autoencoder priors. Advances in neural information processing systems, 34:480 493, 2021.
Rewon Child. Very deep vaes generalize antoregressive models and can outperform them on images. In 9th International Conference on Leurning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.ner 2021. URL htt.:://gganeeeeeee.eee//rrrraaa
Jiali Cui and Tian Han. Learning energy-based model via dual-meme teaching. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Sacnko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Newral Information Processing Systems 2023, NewrIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023. URL httt:://ppppaee mips. nips.cc/ paper
Awwrratt-bbttttt-teee Conference. html
Jiali Cui and Tian Han. Learning latent space hierarchical EBM diffusion models. In Ruslan Salakhutdinov, Zico Kolter, Katherine A. Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.): Forty-first International Conference on Machine Learning, TCML, 2024, Vienna, Austria, July 21-27, 2024, volume 235 of Proceedings of Machine Learning Research, pp. 9633 9645. PMLR / OpenRe- view net, 2024. URL https: httr::://proceedinng mlr. wtt.eer/pee///e/eeeee vatoss///2//cccb4bbbbb cui24b. html
Jiali Cui, Ying Nian Wu, and Tian Han. Learning joint latent space ebm prior model for multi-layer generator. Tn Proceedings of the IEFE/CVR Conference on Compuler Vision and Pattern Recognition (CVPR), pp- 3603 3612, June 2023a.
Jiali Cui, Ying Nian Wu, and Tian Han. Learning hierarchical features with joint latent space energy- based prior. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2218 2227, October 2023b.
Bin Dai and David P. Wipf. Diagnosing and enhancing VAE models. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
URL https://
Yilun Du, Shuang Li, Joshua B. Tenonbaum, and Igor Mordatch. Improved contrastive divergence training of energy-based models. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 2837 2848. PMLR, 2021. URL http://prooceddings. mlr.. .pres//
1099/du2bb.htm
Debottam Dulla, Chaitanya Amballa, Zhongweiyang Xu, Yu-Lin Wei, and Romit Roy Choudhury. Learning energy-based variational latent prior for vaes. XrXiv preprint Xrxi:2250.000 0.00260, 2025.
Ruiqi Gao, Yang Song, Ben Poole, Ying Nian Wu, and Diederik P. Kingma, Learning energy-based models by diffusion recovery likelihood. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.nest net, 2021. URL httpp://oppmmeriie..nffffooo
id=v 1Soh8QUNC
Qiushan Guo, Chuofan Ma, Yi Jiang, Zehuan Yuan, Yizhou Yu, and Ping Luo. Ege: Image generation and classification via a diffusion energy-based model. In Proceedings of the IEEE/CVE International
Conference on Computer Vision, pp. 22952-22962 2023.
Michael Gulmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnor- malized statistical models. In Proceedings of the thirleenth international conference on artificial intelligence and statistics, pp. 297 304. JMLR Workshop and Conference Proceedings, 2010.
Tian Han, Erik Nijkamp, Linqi Zhou, Bo Pang, Song- Chun Zhu, and Ying Nian Wu. Joint training of variational auto-encoder and latent energy-based model. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Song-Chun
13

---

Published in Transactions on Machine Learning Research (02/2026)
Jakob D Drachmann Havtorn, Jes Frellsen, Soren Hauberg, and Lars Maalée. Hicrarchical vacs know what they don't 't know. In International Conference Machine Learning, pp. 4117-4128. PMLR, 2021.
Mitch Hill, Erik Nijkamp, Jonathan Mitchell, Bo Pang, and Song-Chur Zhu. Learning probabilistic models from generator latent spaces with hat cbm. Advances in Neural Information Processing Systems, 35: 928-940, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 00000,68111 2020.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. CoRR, abs/1710.10196, 2017. URL http:://aaw.....g/ab//////////// 10196.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems, 33:12104-12114, 2020.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. Advances in neural information processing systems, 31, 2018.
Alex Krizhevsky, Geoffrey Hinton, Learning multiple layers of features from tiny images. 2009.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. CoRR, abs/1411.7766, 2014. URL http: http://rrxvv.oo//b.b.asa wwwp.as///g..../////////
Lars Maalce, Marco Fraccaro, Valentin Liévin, and Ole Winther. Biva: A very deep hierarchy of latent variables for generative modeling. Advances in neural information processing systems, 32, 2019.
Joanna Marks, Tim YJ Wang, and O Deniz Akyildiz. Learning latent masey-bbaed models via interacting particle langevin dynamics. arXiv preprint arXiv:2510.122 1.122111..1111,, 2025.
Radford M Neal et al. Meme using hamiltonian dynamics. Handbook of markov chain monte carlo, 2(11):2, 2011.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop Deep Learning and Un- supervised Feature Learning 2011, 2011. URL http://ufid] stanford stanfo/r http://t.....aaooeeoeeeoooee//ea
housenumbers. .pdf
Erik Nijkamp, Mitch Hill, Song- Chun Zhu, and Ying Nian Wu. Learning non-convergent non-persistent short-run meme toward energy-based model. Advances in Neural Information Processing Systems, 32, 2019.
Erik Nijkamp, Bo Pang. Tian Han, Linqi Zhou, Song- Chun Zhu, and Ying Nian Wu. Learning multi- layer latent variable model via variational optimization of short run meme for approximate inference. European Conference on Computer Vision, pp. 361-378. Springer, 2020.
Bo Pang, Tian Han, Erik Nijkamp, Song-Chun Zhu, and Ying Nian Wu. Learning latent space energy-based prior model. Advances in Neural Information Processing Systems, 33:21904-22008 22008, 2020a.
Bo Pang, Erik Nijkamp, Jiali Cui, Tian Han, and Ying Nian Wu. Semi-supervised learning by latent space energy-based model of symbol-vector coupling. arXiv preprint arXim:2010.0053: 09359, 2020b.
Benjamin Rhodes, Kai Xu, and Michael U Gutmann. Telescoping density-ratio estimation. Advances in neural information processing systems, 33:4905-49116 2020.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
14

---

Published in Transactions on Machine Learning Research (02/2026)
Tobias Schröder, Zijing Ou, Jen Lim, Yingzhen Li, Sebastian Vollmer, and Andrew Duncan. Energy dis- crepancies: score-indeeendent loss for energy-based models. Advances in Neural Information Processing Systems, 36:45300 45338, 2023.
Casper Kaae Sonderby, Tapani Raiko, Lars Maaløe, Soren Kaae Sønderby, and Ole Winther. Ladder vari- ational autocncoders. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (ods.), Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 201 URL https: 7/proceedings. www.ips.eerpp...m pttpscccaaaperr0///flleeee http://///...aaaaaaaa/e/////////
-Ppperrr pdf
Casper Kaae Sønderby, Tapani Raiko, Lars Maalee, Søren Kaae Sonderby, and Ole Winther. Ladder varia- tional autoencoders. Advances in newral information processing systems, 29, 2016b.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Aduances in Newral Information Processing Systems, 32, 2019.
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. Advances in Newral Information Processing Systems, 33:19667- 2020.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. Advances in Neural Information Processing Systems, 34:11287 11302, 2021.
Zhisheng Xiao and Tian Han. Adaptive multi-stage density ratio estimation for learning latent space energy-basod model. In Sanmi Koyejo, S. Mohamod, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, Novem- ber 28- December 9, 2022, 2022. URL http://papers. www..ic.pppp.p.eeee cc/paperpppppppccpppcpp http://pppepeleeepppppppeeeeee filles//pperr//22//mell 974448892209b00r1112118444400000 -ttpr//aaaaataattaaoooaaaooooo--
html
Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat. VAEBM: A symbiosis between variational autoencoders and energy-based models. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 8-7, 2021. OpenReview.net, net, 2021. URL https://opennevieer hee/fforum11d=6m33EEezZ0V
Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu. A theory of generative convnet. In International Conference on Machine Learning, pp. 2635-2644. PMLR, 2016.
Xuwang Yin, Shiying Li, and Gustavo K. Rohde. Analyzing and improving generative adversarial training for generative modeling and out-of-distribution detection. CoRR, abs/2012.06568, 2020. URL https:
//arxiv. wttaaa/.........
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. LSUN: construction of largo-scale image dalaset using deep learning with humans in the loop. CoRR, abs/1506.0333 2015. URL http:
7arxiv org/ abs/ 1506. 03366
P Yu, S Xie, X Ma, B Jia, B Pang, R Gao, Y Zhu, S-C Zhu, and YN Wu. Latent diffusion energy- based model for interpretable text modeling. In International Conference on Machine Learning (ICML 2022).. 2022.
Peiyu Yu, Yaxuan Zhu, Sirui Xie, Xiaojian Shawn Ma, Ruiqi Gao, Song-Chun Zhu, and Ying Nian Wu. Learning energy-based prior model with diffusion-amortized meme. Advances in Neural Information Pro- cessing Systems, 36, 2024.
Shiyu Yuan, Jiali Cui, Hanao Li, and Tian Han. Learning multimodal latent generative models with energy- based prior. In European Conference on Computer Vision, pp. 86-100. Springer, 2024.
A Additional Result
Joint v.s. progressive sampling.
15

---

Published in Transactions on Machine Learning Research (02/2026)
We compare the two strategies quantitatively. As shown in T progressive sampling can yield slightly better generation quality, but at a much higher computational cost (in total 30 steps Langevin dynamicss) since each stage requires sepa- rate Langevin updates. Joint sampling, contrast, delivers strong performance with far greater effi- ciency (in total 10 steps Langevin dynamics), mak- ing it the default strategy in our framework. This analysis confirms that our multi-stage hierarchical EBM prior supports both flexible stage-wise refinement and efficient joint sampling.
Table 7: FID for image synthesis of different stages.
Prgrogressive Sampling
Joint Sampling
Stage
Stage
Stage
CIFAR-10
13.15
9.72
7.62
7.80
LSUN- ChuN-hhurb--64
14.82
10.06
6.12
6.65
CelebA-H1Q-256
11.48
8.65
7.71
7.84
Stage-wise Analysis with Diffusion Scheme. Both diffusion based EBMs Cui & Han 2024 and our multi-stage contrastive framework progressively refine the prior distribution by decomposing the modeling task into simpler intermediate steps. T compares the generation quality (FID) of intermediate stages.
While both approaches improve with more refinement stages, method shows clear advantages. Diffusion- based models rely fixed schedules and Gaussian perturbations, which restrict flexibility and may lead to inefficient sampling. In contrast, our multi-stage framework is self-adaptive where cach stage learns the density ratio between its evolving prior and the target posterior, which ultimately yields better generative quality.
Table 8: Comparison of diffusion-based and multi-stage priors across refinement steps. t t= (2, 1, denotes diffusion steps; (1,2,3) denotes stages in our multi-stage hierarchical EBM prior.
FID
Gaussian Prior
{ == s=1
t= 1
8 2
t= 0=0/ / == 3
CIFAR-10
37.73
18.11
13.15
11.87
9.72
8.93
7.62
LSUN- ChUN-Chrrh-661
38.13
20.15 / 14.82
13.54
10.06
7.31
6.12
SelebA-HQ-225
30.25
17.75 / 11.48
12.05
/ 8.65
8.78
7.71
Sample Quality Evolution. To provide additional evidence of training stability, we track the evolution of sample quality measured by FID. Specifically, we report FID at regular checkpoints both within each stage and across stages. We compute FID every 2,000 iterations, and every 5,000 itcrations we transition to the next stage of learning (denoted by 1, 3).. The results on CIFAR-10 and LSUN-Church664 are summarized in Table g As shown, FID decreases as learning progresses, and importantly, there is no degra-
Table 9: Evolution of generation quality FID (L) over iterations and stages.
2
8 3
2000 iter
4000 iter
6000 iter
8000 iter
10000 iter
12000 iter
best
CIFAR-10
21.45
14.34
12.85
10.55
9.66
7.84
7.62 (13500 iter)
LSUN-Chureh-61
19.61
15.87
11.22
10.10
10.03
7.10
6.12 (14500 iter)
dation or fluctuation in quality when switching to the next stage. This improvement across itcrations and stages indicates that the dynamic loss behavior observed in Fig accompanied by consistently improving generation quality.
Impact of Network Structure and Decoder Type. In our default configuration, the NVAE backbone employs a discretiizd logistic decoder. To further evaluate the robustness of our approach, we also experiment with the NVAE backbone using a Gaussian decoder and network structures, consistent with the setting in NCP-VAE Aneja et al. (2021). As shown in Tab our multi-stage learning consistently outperforms the single-stage baseline, achieving substantially lower FID scores. These results demonstrate that the benefits of our multi-stage framework are not tied to specifie decoder choice, but generalize across network structures.
16

---

well Langevin transition results in Published in Transactions on Machine Learning Research (02/2026)
Table 10: Effect decoder type on C1
CIFAR-10
Gaussian prior
Stage
Stage
NCP-VAE
52.45 52.45
24.08
Ours
20.68
14.04
We show additional image synthesis in Fig 11
Fig 12
and Fig
13
Fig
14
B
Derivation
B.1
Joint Learning Objective
Learning hicrarchical generative models (Eqn. 9 can be achieved by maximizing the log-likclihood as Cp(3, wim) N E Pa... (X;), where Paaum (x:) -2 Pa.c.. parameters 3, the gradient is computed as
Z)dZ. Z).Z. For learning the generator
03 C3 wm) = E Prisia am (Z
log Pp.wm (X, Zi)
03
8
Passa (aannnX)pppaa (X)pa.um (2(X)
log P%, (X|Z)
+ PBaaata (X)aat(X)pps.a (Z)x)
log Pur 300 (2)
(15)
a
0g
i
i
EEaasa Paaas(X))aa... (X)pa.um (2(x)
log Pd. (
+ Epaata (N) (Z)x
log P330 (Z)
+C
a
09
where C is constant with respect to 33.
For learning the EBM prior Wm, we adopt contrastive estimation Gutmann Hyvärinen (0110). i.e. mini-
mizing (z) log Dum (Z)] (log) Du... (Z)),, which corresponds learning density-ratio estimator
Twm (Z) p(Z) --rm
g(Z) Dem (2) The reparameterized objective yields the learning gradient as
i Duim
Ey(z) (log Dwm. (Z)) +
(pzz)/oo(( Dum (Z))]
(16)
I Owim
Tum (Z)
② Owim
1
(2))
log
+ Ep())
log
+ Fuim (Z)
Twom
ZZ)
Here, we consider the target distribution ((Z) as the generator posterior from the previsous stage, i.e., Pdata (X)psm- ()()), and p(Z) is the hicrarchical EBM prior from previous stage well, i.c., p(Z)
Pumm 1. ().. These derivations lead to our joint-training objectives in Equ. 10 and Equ. 11
B.2 Two-phase Learning Objective
In the first-phase learning, we adopt variational scheme for the hierarchical latent variables by rotruigg hierarchical inference model 9o (x) and maximizing the corresponding ELBO (Eqn. [2]. This ELBO is equivalent to minimizing the KL divergence between two joint distributions, the data inference joint Pdaza (X)g (2 and the hierarchical generator joint pa (X, Z), as
DKL (X) (Z (X, Z))
X)) + E
(17)
In this decomposition, the first term encourages the generator to match the data marginal over X, while the second term encourages the variational posterior go (Z(X) to approximate the true generator posterior Pa (Z(X). As such, we can efficiently draw approximate posterior samples.
Specifically, in the socond-phase learning, we treat the posterior samples obtained from the first phase as defining the target distribution in the latent space. Concretely, we draw Z (a(Z(X) with X Paata from the learned first-phase model. If the first-phase model well-learned, the variational posterior is close
17

---

Published in Transactions on Machine Learning Research (02/2026)
to the generator posterior. This empirical assumption is supported by the strong reconstruction quality shown in Tab 3 In this regime, the induced target distribution over latents can be viewed as the aggregated
posterior
(Z) f 9o (Z) (Paa) (X)p P Paata Pdata (X)XX
(18)
However, this aggregated posterior is typically complex and exhibits a substantial mismatch with the simple Gaussian prior, phenomenon often referred to as the prior-hole problem Aneja et al. (2021); Cui & Han
h2024h. The goal of the second phase is therefore to learn hierarchical EBM prior Pw,0so (Z) that matches the aggregated posterior and bridge this gap. Maximizing the log- bigliikeihood under the EBM reduces to minimizing the KL divergence:
arg max (q(Z) Por (Z) = arg max - -By(z) log "
+ C = arg min DKI. (kI,)g())llw),,.. (Z) (19)
Pa.850
(Z)
where C is the entropy term of Zz) [log (Z)) being constant during this phase. Thus, second-phase learning amounts to minimizing this KL divergence (ours Eqn. aligning the hicrarchical EBM prior with the
aggregated posterior distribution.
B.3 Uni-scale Transformation
For conditional Gaussian distribution Pa, (z,(Zz4)), an invertible transformation function T8 be defined Xiao et al. (2021). Take 2-layer latent variables as example,
22 = T500 (W2) = w2 and
(20)
21 = T2 (W1, w2) = ue (22) + a3 (Z2) W1
where W1 and W2 are distributed as independent Gaussian noise, i.e, (W1, W2) ~ Po(W1, w2) and Po(W1, w2) = (o((w))u(ws) with cach po(w,) ~ ~NO0 Ia). By the ofanal-of-variable rule, we have
P3 (2) Po Po w2) det(S ) and
(21)
Po Po(W1, W2) = P3s. (2), 22) det (et((Jsso))
where JTpy is the Jacobian of T
For joint EBM prior Z-space Pa, (Z) -w (Z))ps88 (Z), we apply the change-of-variabbe rule and
Eqn. 21 as
Pu.Bzo (W) = Pw. (e)
=ru(Tos ()) ( dct (J
(22)
(Teso Pu) Pu W)
With such single-stage EBM prior on W-space, we construct our multi-stage EBM prior.
MCMC Posterior Sampling. To obtain samples from the generator posterior (i.e., Pu,a (Z(X) x
Pao (X, (X,Z)) Z)), the target distribution the uni-scale W-space is derived as
Pa X. W) = Pa. (X ()) (e) )l = Ps, T9so (W)) (W)
(23)
Comparison with Z-space. In the original Z-space, each z, can reside in different scale. In particular, for layers i # j, the conditional priors Pa. (z./241) and Ps, (z, [z)+1) may have distinct parameterization {2 and {Ma, This induces variations in the magnitude of different scales of z, and Zj, result- ing in a multi-scale latent space. Sampling in such space can be challenging: a single global step size is often suboptimal, and one may need carefully designed, scale-adapted updates to ensure stable and efficient exploration. In contrast, our W-space is uni-scale by construction: all w variables are drawn from a fixed, standard Normal distribution. This enforces a consistent scale across layers and dimensions, greatly simpli- fying the design of sampling algorithms, reducing the need for ad-hoc scale corrections, and leading more effective sampling behavior in practice.
18

---

Published in Transactions on Machine Learning Research (02/2026)
B.4 Coupling with Symbol Vector
Inspired by prior advance Salimans el al. (2016); Pang et al. (2020b), extend our hierarchical prior with an additional latent-label coupling module to enable controllable generation. After training the hierarchical generator in a fully unsupervised manner in the first phase, we fix the Icarned generator and learn our hierarchical EBM prior with label vectors y. In this way, our model supports controllable generation without re-training the backbone generator.
Specifically, our joint density can be factorized as Pu (W,)) Pum W Pum..00 (W), where P0o ( () is K-way classifier (i.e., softmax classifier), and Pum. (W) is our hierarchical EBM prior. Such ratio estimator (T (W), v) can be interpreted as classifier over (K+1) classes. In practice, we define our ratio estimator as Tw (T W), which produces K -1 outputs. The first K outputs correspond to classes y = {1... ,,, and the (K + 1)-th output plays the role of distinguishing samples from the base distribution (EBM prior of previous stage) versus the target distribution (aggregated posterior).
C Discussion of Latent Space Modelling and Learning
Latent variable generative model specifies joint distribution with low-dimensional latent space as
Ps (x z) = p3 (x))))oo(z
(24)
where pz( is the prior model usually assumed to be standard Gaussian, and p=(x(z) is the generation model that maps from low-dimensional latent space to high-dimensional data space. However, such a Gaussian prior model can be non-informative and inxxpressive, making the modelling capacity limited and hurting the generative performance.
To tackle this issue, prior arts explore learning hierarchical prior model by factorizing multiple layers of latent variables (Eqn. D. such as
Hierarchical Prior.
L-1
Po(X, Z) Pdo X p)p ( (Z) where P3 (Z)
II i=1
pa. (z, ((z2))
(25)
Such hierarchical generative models or relevant forms have shown strong ability to model complex data distributions and, importantly, to learn multi-level latent representations that reflect semantic hicrarchies in the data Sønderby et al. (2016b); Nijkamp et al. (2020): Vahdat & Kautz (2020); Child (2021). However, most of these models still assume each conditional prior to be Gaussian, which primarily captures dependencies across layers, while ignoring the intra-layer relationship, resulting in the hicrarchical latent space being under-modelled.
Ensrgy-based Prior. Alternatively, the latent variable generative model with EBM prior (LEBM) Pang ct al. 2020a) specified joint distribution
Paw X. = (x(
(26)
(z) = 2 exp (z) becomes the EBM prior, which can be more expressive than a Gaussian prior. However, learning EBM prior by MLE requires MCMC sampling, such Langevin dynamics Neal et al.
(2011), as an inner-loop during the training. This motivates follow-up works of different modelling schemes
for EBMs (c.g., score-matching Guo et al. (2023)) and lcarning schemes (c.g., energy discrepancy Schröder et al. 2023), NCE learning Rhodes et al. (2020): Xiao & Han (2022)) )
Hierarchical EBM Prior. More recently, advances also explore hierarchical EBM prior Cui et al. 2023a Aneja et al. (2021); Cui et al. (2023b) which intend to leverage the strengths of both the hierarchical generative model and FBM prior. In this setting, the joint distribution is specified
Pa (X, = pa (x (Z)
(27)
where Pw.Bso (Z) becomes hicrarchical prior modulated by EBM. However, the aggregated posterior is typically highly multi-modal and strongly structured, and learning a single-stage EBM to approximate the aggregated posterior is difficult, leading to suboptimal learned hierarchical EBM prior.
19

---

Published in Transactions on Machine Learning Research (02/2026)
Multi-stage learning. To address the limitations of single-stage learning, multi-stage EBM frameworks have been proposed. One branch of this line of work is diffusion-based EBM priors Gao et al. 2021 |; Guo et al. (2023); Cui & Han (2024); Schröder et al. (2023). They model a sequence of gradually perturbed distributions (from simple noise to the target distribution) and learn score functions along a pre-defined
noise schedule. These approaches are powerful but typically rely on carefully designed schedules and can be sensitive to hyperparameters such as the number and spacing of noise levels. Another branch multi-stage NCE, which offers self-adaptive scheme without fixed schedule. Specifically, Rhodes al. (2020) uses fixed number of stages in data space, but far shows convincing results only relatively simple datasets like MNIST. Xiao & Han (2022) introduces adaptive multi-stage schemes in flat latent spaces, but still does not realize the hierarchical structure in the latent representation. Our work builds on this line of research by introducing a multi-stage hierarchical EBM prior.
Broader Impact Statement
This paper proposes generative probabilistic framework aimed at advancing hierarchical representation learning and controllable gencration, and thus shares similar potential risks as other powerful generative models. In particular, fine-graine nre-grained control over semantic attributes could, in principle, be misused to create manipulated or mislcading visual content. At the same time, our work on OOD detection may help mitigate some risks by enabling more reliable detection of anomalous or out-of-distribution inputs in downstream systems.
E Implementation and Algorithm
E.1
Implementation
Training detail. For joint learning, we benchmark our method on CIFAR-10, SVHN, and CelebA-64, for which we scale training images to 1] and use only 40,000 examples of CelebA-64 following Xiao & Han (2022). For Two-phase training, we train our model on CIFAR-10, L.UUNChurch-64, and CelebA-HQ-256 using the same setting as NVAE Vahdat & Kautz (2020). We compute FID scores using 30,000 generated images for CelebA-HQ-255 -4Q- and 50,000 for other datascts.
We use one A100 Nvidia GPU (Iraining time CIFAR-10 for 0.15 reconds/iterationn for joint learning and two A100 Nvidia GPUs (training time CIFAR-10 for 8.45 seconds /ittertttoonn for Two-phase training, Please see further implementation details at https://github httpgitttuu.c..mae.eee http://cwil////1///1
nce- ebm
Network Structure. For joint learning, we adopt the network structures from Xiao & Han (002),, which contain generator and energy function networks shown in Tab[12] For Two-phase learning scheme, we utilize the NVAE? backbone model, and our energy function shown in Tab 11
20

---

Published in Transactions on Machine Learning Research (02/2026)
E.2
Training Algorithm
Algorithm Joint Learning Scheme
Require:
Training images x; Training iteration i; Each stage iteration 7; Current stage number my Total stages
M; Joint EBM prior wmi Hierarchical generator model 3; Posterior Buffer Bm and Bm-1; Langevin
steps k;-
Let m 0, initialize wm and Bm.
repeat
repeat
4: Posterior Sample: obtain W Wtrue Pw .f (Z|X) using Equ 14 with k.
5: Save Buffer: Save posterior sample W true to Posterior Buffer Bm-
6: Prior sample: Draw Wake from Bm-1 if m > 0 clse Wiake ~ (N.
7: Learn 3mt Update fm with Wrree using Zerue T3 (W true) and Equ. 10
Learn Wmi Update Wms with Ztrue (left term in Eqn. and Ztake Ta W take) (right term in
Eqn. 11
until =
10: Replace Bm by Bn and initiate new By
11: until m = .
Algorithm Two-phase Scheme
Require:
Training images x; First-phase training iteration d1; First-phase total iteration I;; Hierarchical model
B, Second-phase training iteration i2; Each stage iteration Iz; Current stage number m; Total stages
M; Joint EBM prior Wm; Langevin steps k; Prior Buffer B;
1: (1) First-phase learning: initialize 3, d.
2: repeat
3: Variational learning for 8, update 3, via Eqn.
until i=
(2) Second-phase learning: Fix 3. and initialize Wm.
repeat
if m 0, prepare Buffer Bm-1 by EBM prior sampling via Equ 14 with else continue.
8: repeat
9:
Posterior Sample: obtain Zurue ~ (Z) and Wurue = 33 (Zuree
10:
Prior sample: Draw Wiaee from Bm-1 if m > 0 else Wiaker N 7)
11:
Learn Wm: Update wm with Ztrue T (W (Wrre) (left term in Eqn. 13 and Ziake = TB2o Winke)
(right term in Eqn. 13
12: until i2 = Iz
13:
until m M
Figure 11: Additional synthesis on LSUN-Church-64.
21

---

Published in Transactions on Machine Learning Research (02/2026)
Figure 12: Additional synthesis on CIFAR-10.
it
V
Figure 13: Additional synthesis on CelebA-HQ HQ-256 with temp= 1.0 (left) and temp= 0.7 (right).
Figure 14: Additional Langevin transition on 65UN--Curch-64, CIFAR-10, and CelebA CelebA-HQ-256.
22

---

Published in Transactions on Machine Learning Research (02/2026)
Figure 15: Additional progressive Langevin transition Church-6,, CIFAR-10, and CelebA-HQ- 256.
Table 11: Two-phase learning energy function on each z,
Table 12: 2-layer generator network on CIFAR-10.
Coay Resicule Blodk (Ineh, ont-ch, dounasmple, proActivationn
Tapest: SILUU(s) Aetiosaato else Ag(ha) (xl conv (ont- chi,, s=2, SSLU
if downsmplle E nos downsample if tn-thh-out-ch downssmpple if downample if in chypout ch
(x(()))) ngf 64
Ri(h)) axd cone ch), s-1, SILU
(ous-ch) haue conv (ont-ch), 2=2
ky(h,
Layers
In-Out Size
Stride
Inpat: x
hata (Da) lad conv (out ch) s-1
4x4 conyTT (ogf 8), LReLU
4
(ngf 8)
return by - Jubust
fxi convTT (agl 1), LReLU
8 x (ngf 1)
2
4xi con'TI (agf 2), LRtLU
16 16 (ngf
2
Cour Residule
4x44 conTT (3), Tanh
32 x 32 x
2
(Irwch, oatlch,, wa-loyer)
Phi (R1(2s), ogl= 256
Topec:
Cour Reuidule Black (in th, out-ch, dorasscople= True, preActivation Trae) #res layer x Cany Residule Block (In-ch, out-ch. downsample- False, co.Activaton-Thee
Layers
In-Out Size
Inpat: Rg
100
Lineaz Residule Black
Linear (ngC, LReLU
ngf
(lneh,, out-ch, preActisatioo)
Liuear (ngC, LR=LU
ngf
Topert: Si1UU(g) perAeeivoation else by
(ngf). LReLU
ngf
6itha) Linear (out cb), SILU (2,)) Linear (ous-ci)
Linear (ngf), LReLU
ngf
in-chyoout-ch in-dheoonteh else
Tincar (100), y of wi
100
haver Liswar (ont-ch)
Linear (100). logo
100
emergy function for each w., ndF
Energy function
Layers
In-Out Size
(nef-129, ndi-256, ronelgee--)
Inpul:
100
Layer
In Out Size
Linear (ugf), LReLU
ndf
Iaput:
E ch nof
Linear (ngf). LReLU
ndf
come (nce), am1
Tinear (ngf, L.R..U
ndf
# of Cony Rosidule (nef, nef, res-lager) Batten
xxx 4 x 64
Linese (ngf) LReLU
ndl
4 4 64
Linear (1)
1
* res-layer of Linent Residule (adf, udf, pecActivation True) SIAU, Linear (1)
23


Reviews:
[{"review_id": "eQf5zufIh1", "reviewer": "Reviewer_xRFw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a novel framework that introduces multi-stage energy-based models (EBMs) priors for hierarchical generative models. The authors claim that this framework can enable more expressive latent modeling than a Gaussian or single-stage EBM prior. To improve the effectiveness and efficiency, the authors use a uni-scale latent space reparameterization and develop two training schemes.", "claims_and_evidence_yn": "Yes", "claims_explanation": "This paper provides a detailed explanation of the method they proposed, and presents visual effects and quantitative comparisons in the paper.", "audience_yn": "Yes", "audience_explanation": "This paper introduces a multi-stage energy-based prior for hierarchical generative models. It provides valuable insights for subsequent researchers, both in enhancing the expressiveness of latent modeling and in designing more efficient generation pipelines.", "requested_changes": "There are some concerns and questions:\n\n1. The author only conducted experiments on small-scale datasets such as CelebA-HQ-256, LSUN-Church-64, and CIFAR10. At present, the generation models need to undergo experiments on ImageNet, such as the papers like MAR, SiD, and DiT. So, could the author provide an experimental result on ImageNet?\n\n2. Considering that this paper was submitted in late 2025, the most recent baseline methods included in the comparisons were published in 2024. This may somewhat limit the assessment of the proposed method’s effectiveness. Could the authors consider adding comparisons with more recent methods?\n\n**I am not an expert in the field of Hierarchical Generative Model. For the novelty of this paper, please refer to the opinions of other reviewers.**", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kqzIyL7uMj", "reviewer": "Reviewer_GZj5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper tackles the \"prior-hole\" problem in hierarchical generative models by introducing a Multi-Stage Hierarchical Energy-Based Model (EBM) Prior. The proposed framework is mathematically solid, effectively decomposing the difficult density ratio estimation task into a sequence of adaptive, manageable stages. The authors also introduce a technically sound \"uni-scale\" reparameterization to address the challenges of multi-scale latent spaces. The experimental results are promising, demonstrating that this mathematically grounded approach consistently improves generation quality over single-stage baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The experimental section is comprehensive and the results are promising. The comparisons against various baselines (including single-stage EBMs and diffusion models) provide good evidence that the multi-stage refinement strategy is effective. The ablation studies further support the mathematical intuition behind the stage-wise decomposition.", "audience_yn": "Yes", "audience_explanation": "The paper presents a solid theoretical contribution to the intersection of VAEs and EBMs. The rigorous mathematical formulation of the multi-stage density ratio estimation in hierarchical spaces will be of significant interest to researchers focusing on rigorous probabilistic modeling and latent space dynamics.", "requested_changes": "I recommend the following adjustments to strengthen the submission:\n\n1.  **Evidence of Stability in Joint Training:** The paper emphasizes the stability of the joint learning scheme and points to the smooth loss trends in Figure 8 as evidence. However, in contrastive or adversarial settings, a stable training loss does not strictly guarantee better generation quality (e.g., it could indicate mode collapse). Can the authors provide further evidence of stability beyond the loss curves? For instance, monitoring the variance of FID scores across different training checkpoints or providing visual samples over time would be much more convincing evidence that the \"stable\" loss correlates with stable generation capability.\n2.  **Clarification on Controllability:** In Section 5.2.1, the paper discusses \"Hierarchical Controllability\" and mentions extending the EBM into a multi-class classifier using \"K class labels.\" However, the text also claims the backbone generator is trained in a \"fully unsupervised manner.\" This creates confusion regarding the supervision setting.\n    * How exactly are the concepts of gender or attributes \"injected\" into the model?\n    * Does the EBM training require labeled data while the generator does not?\n    * If labels are used, please clarify this distinction to ensure the claims about unsupervised learning are accurate and the method's data requirements are clear.", "broader_impact_concerns": "The authors discuss \"Fine-grained Controllability,\" which allows for the manipulation of specific attributes like gender or appearance. While this is a technical contribution, it raises standard concerns regarding the generation of manipulated media. I request that the authors include a brief Broader Impact Statement acknowledging the potential misuse of such controllability features, alongside the positive applications of their OOD detection work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DpWcOzZha2", "reviewer": "Reviewer_3mes", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a framework for learning multi-stage hierarchical Energy-Based Model (EBM) priors to resolve the prior-hole problem in hierarchical generative models (HGMs). Instead of relying on a single, less expressive Gaussian prior, the method uses a sequence of density-ratio estimators to progressively refine the EBM prior and match the complex, multi-modal aggregated posterior. The approach achieves superior generative performance and yields richer hierarchical representations, enabling both generative modelling and out-of-distribution detection.\n\nAlthough most of the techniques employed in this work are based on prior studies. For example:\n\n- the multi-scope NCE approach from [1],\n- the “uni-scale” sampling trick for latent EBMs from [2]\n- the multi-stage latent EBM training strategy from [3].\n\nThe integration of these methods to scale EBMs for image modeling and achieve performance competitive with diffusion models makes this paper promising.\n\n[1] Rhodes, Benjamin, Kai Xu, and Michael U. Gutmann. \"Telescoping density-ratio estimation.\" *Advances in neural information processing systems* 33 (2020): 4905-4916.\n\n[2] Xiao, Zhisheng, et al. \"Vaebm: A symbiosis between variational autoencoders and energy-based models.\" arXiv preprint arXiv:2010.00654 (2020).\n\n[3] Xiao, Zhisheng, and Tian Han. \"Adaptive multi-stage density ratio estimation for learning latent space energy-based model.\" Advances in Neural Information Processing Systems 35 (2022): 21590-21601.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims are supported by extensive and well-structured experimental evidence.", "audience_yn": "Yes", "audience_explanation": "Yes. Researchers working on EBMs will likely appreciate the proposed scalable multi-stage learning paradigm. It is encouraging to see that EBMs can be effectively scaled for image modeling. Although the methodology largely integrates existing techniques, the resulting strong performance is still impressive and valuable to the community.", "requested_changes": "- The justification for the objective function in Equation 4 could be strengthened. It wasn't immediately clear what the theoretical connection is between this equation and the principle of maximum likelihood. A detailed discussion in the appendix would be a valuable addition.\n- The term \"uni-scale\" introduced in Section 2 requires clarification. While I suspect this terminology may be used because the variable $W$ resides in a Gaussian noise space, the paper would benefit from a more detailed and explicit explanation of what \"uni-scale\" means and why this property is relevant.\n- I would suggest including the derivations for Equations 11 and 12 in the appendix to make the paper more self-contained. The paper [4] (see Appendix C.1) provides a very detailed discussion on various methods for training LEBMs (e.g., CD, NCE, SM). Incorporating a similar, brief discussion of these training paradigms would be helpful for the reader.\n- In Equation 14, the posterior p(Z|X) is replaced with a variational distribution, but the EBM prior p(Z) still requires MCMC sampling. It would be helpful to explicitly clarify this point in the paper.\n- Following up on this observation, it is unclear why only p(Z|X) is replaced with a variational distribution, while p(Z) is still modeled via MCMC. Have the authors considered or conducted an ablation study comparing the following configurations:\n    - Both p(Z|X) and p(Z) modeled with variational approximations\n    - Both p(Z|X) and p(Z) modeled with MCMC\n    - Variational for p(Z|X) but MCMC for p(Z) (the current approach)\n    - Variational for p(Z) but MCMC for p(Z|X)\n- It would also improve clarity if the paper included a concise algorithm summarising the overall training and inference procedures.\n\n[4] Schröder, Tobias, et al. \"Energy discrepancies: a score-independent loss for energy-based models.\" *Advances in Neural Information Processing Systems* 36 (2023): 45300-45338.", "broader_impact_concerns": "There are no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)