Hierarchical Semi-Implicit Variational Inference with Application to Diffusion Model Acceleration
Authors: Longlin Yu
Tianyu Xie
Yu Zhu
Tong Yang
Xiangyu Zhang
Cheng Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: Hierarchical semi-implicit variational inference
Score based training
Diffusion model
TL;DR: A novel semi-implicit variational inference method with hierarchical structure, which shows promising results on diffusion model acceleration.

Abstract: Semi-implicit variational inference (SIVI) has been introduced to expand the analytical variational families by defining expressive semi-implicit distributions in a hierarchical manner. However, the single-layer architecture commonly used in current SIVI methods can be insufficient when the target posterior has complicated structures. In this paper, we propose hierarchical semi-implicit variational inference, called HSIVI, which generalizes SIVI to allow more expressive multi-layer construction of semi-implicit distributions. By introducing auxiliary distributions that interpolate between a simple base distribution and the target distribution, the conditional layers can be trained by progressively matching these auxiliary distributions one layer after another. Moreover, given pre-trained score networks, HSIVI can be used to accelerate the sampling process of diffusion models with the score matching objective. We show that HSIVI significantly enhances the expressiveness of SIVI on several Bayesian inference problems with complicated target distributions. When used for diffusion model acceleration, we show that HSIVI can produce high quality samples comparable to or better than the existing fast diffusion model based samplers with a small number of function evaluations on various datasets.

Hierarchical Semi-Implicit Variational Inference with Application to Diffusion Model Acceleration
Longlin Yu!.*, Tianyu Xiel,", Yu Zhu Tong Yang", Xiangyu Zhang", Cheng Zhang 1.2.
School of Mathematical Sciences, Peking University
2 Center for Statistical Science, Peking University
Institute of Automation, Chinese Academy of Sciences
Beijing Academy of Artificial Intelligence
Megvii Technology Inc.
{llyu, Semryuxxiepppikee edu. cn, zhuyu20220ia. ac. en,
Kyangtong, zhangxiangyu Amegvii com, chengzhang@math. www.ch..kkuuuuu.... edu. cn
Abstract
Semi-implicit variational inference (SIVI) has been introduced to expand the an- alytical variational families by defining expressive semi-implicit distributions a hierarchical manner. However, the single-layer architecture commonly used in current SIVI methods can be insufficient when the target posterior has complicated structures, In this paper, we propose hierarchical semi-implicit variational infer- ence, called HSIVI, which generalizes SIVI to allow more expressive multi-layer construction of emmiiitplicit distributions. By introducing auxiliary distributions that interpolate between a simple base distribution and the target distribution, the conditional layers can be trained by progressively matching these auxiliary dis- tributions one layer after another. Moreover, given pre-trained score networks, HSIVI can be used accelerate the sampling process of diffusion models with the score matching objective. We show that HSIVI significantly enhances the expressiveness of SIVI on several Bayesian inference problems with complicated target distributions. When used for diffusion model acceleration, we show that HSIVI produce high quality samples comparable to better than the existing fast diffusion model based samplers with a small number of function evaluations on various datasets.
Introduction
Variational inference (VI) is approximate Bayesian inference method that is gaining in popularity, where one tries to find an approximation to the target posterior distribution using an optimization approach (Jordan et al. 1999: Wainwright & Jordan 2008 Blei al. 2016). To do that, it first posits a family of variational distributions and then seeks the closest member from this family that minimizes some statistical distance to the target posterior, usually the Kullback-Leible (KL) divergence. As the posterior is not analytically available, an equivalent formulation is often adopted in practice where maximizes the evidence lower bound (ELBO) instead (Jordan al. 1999).
One classical VI method is mean-field VI, which assumes a factorizable structure of the variational distributions over the parameters latent variables (Bishop & Tipping 2000). This often leads to closed-for form coordinate-asces update rules when certain conditional conjugacy conditions are satisfied. In practice, the conditional conjugacy may not hold and the true posterior could be much
'Equal contribution. This work was done during internship at Megvii Technology CCorresponiing Author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

more complicated than what a factorized variational distribution can accurately approximate. In recent years, several attempts have been made in VI that alleviate these constraints by designing more
flexible variational families (Jaakkola & Jordan 1998; [Saul & Jordan 1996. Giordano et al. 2015;
Tran et af.] a 2015] Rezende & Mohamed| [015] Dinh et al. 2017 Kingma et al. 2016; Papamakarios
et al.
2019). together with generic training a algorithms via Monte Carlo gradient estimators (Nott
ct al.
2012, Paisley et al., 2012 Ranganath et al., 2014, Rezende et al. 2014 Kingma & Welling
2014p.
While successful, these approaches all assume tractable densities of variational distributions.
To further expand the capacity of variational families, one approach is to incorporate the implicit
models that have intractable densities but are easy sample from (Huszár 2017] Tran al. 2017;
Mescheder et al. 2017: [Shi et al. 201 b. Song et al. 2019). 19. However, as the densities are intractable for implicit models, one often resorts to density ratio estimation for ELBO evaluation during training,
which is known to be difficult in high dimensional settings (Sugiyama et al. 2012). ) To avoid density ratio estimation, semi-implicit variational inference (SIVI) has been proposed where the variational distributions are formed through a semi-implicit hierarchical construction, and various training criteria
have been employed (Yin & Zhou 2018: Moens et al. 2021 Titsias & Ruiz 2019 Yu & Zhang 2023).
While striking a good balance between approximation flexibility and training efficiency, current SIVI methods often use a single conditional layer which can be insufficient when the target posterior possesses complicated structures (e.g., multimodality, see an example in Section 5.1p. To enhance the expressiveness of single-layer models, an intuitive but effective approach is to extend them to multi-layer hierarchical models (Vahdat & Kautz, 2020; Ranganath et al. 2016: (Sobolev & Vetrov 2019). In this paper, we propose hierarchical semi-implicin variational inference (HSIVI), which is a generalization of SIVI that allows multiple conditional layers. Instead of training the hierarchical semi-implicit model end to end, we introduce auxiliary distributions that interpolate between a simple base distribution and the target distribution to guide the intermediate semi-implicit distributions toward the target distribution. The conditional layers are then trained sequentially to match these auxiliary bridging distributions given the fitted semi-implieit distributions from the previous layers (Figure D. using different criteria from before. This way, HSIVI allows progressive learning of the target distribution that significantly reduces the burden of each conditional layer. Moreover, HSIVI with the score matching objective can also be used to accelerate the sampling process of diffusion models where the pre-trained score networks corresponding to different noise levels provide a natural sequence of bridging distributions. In experiments, we demonstrate the effectiveness of HSIVI on both Bayesian inference tasks with complicated target distributions and diffusion model acceleration.
2 Background on semi-implicit variational inference
The semi-implicit variational family (Yin & Zhou 2018 Titsias & Ruiz 2019) is defined as
(x - 4o (z))d))
(I)
where á are the variational parameters, go (xx(2) is called the conditional layer, and g(z) is called the mixing layer. This variational family is said be semi-implicit as go (2(2)22) is required to be explicit and q(x) is often implicit. The semi-implicit variational family is capable of capturing more complicated dependencies between variables (Yin & Zhou [0118) Titsias & Ruiz 2019:] Yu & Zhang 2023) than explicit variational families without the hierarchical structure. Given the observed data D, the classical VI methods often use the evidence lower bound (ELBO) for training, which is defined as ELBO- := (x) [logr( D, x) - log (æ)]. However, as g (x() is no longer tractable in SIVI, alternative training objectives have been introduced.
ELBO related objectives
Yin & Zhou 2018) considered a sequence of lower bounds of the ELBO
p(D, x)
(swvas(p((=/D))lloeeee) = E. (=)
log
(200 y bid (id.
a(s)
(0(2) - EK 2o (z((((())
R+1
is an asymptotically exact surrogate in the sense that link- LSIVLLIB = ELBO. Titsias& & Ruiz (2019) proposed unbiased implicit variational inference (UIVI) which uses samples from the inverse conditional distribution go (z) a²) (from an MCMC run, e.g. Hamiltonian Monte Carlo (Neal) 2011) to provide unbiased gradient estimator of the exact ELBO. See more details of UIVI in Appendix B

---

Pstra
9ul% (as)
Aprin (
Conditional
Auxillory distribution
Bose distribution
Figure I: An example for 4-layer HSIVI. The target distribution po(x) a Gaussian mixture and the auxiliary distributions {p:())) are constructed using the diffusion bridge. The auxiliary distributions are plotted in the squares, where the blue heatmap describes the probability density and the arrows represent the score functions of the auxiliary distributions.
Score matching objective Besides the ELBO, score based distance measures have also been used for variational inference where the score function S(x)) (r) := V x log p( D) Var log D. .x) is
assumed to tractable (Liu al. 2016 Zhang et al. 2018: Hu al. 2018). Yu Zhang (2023)
considered the following Fisher divergence between the target distribution and the semi-implicit variational distribution
Drisher (p( (D)lus (x)) [((x) V log (x)
By reformulating Drisher as the maximum of the following optimization problem
Drshher (aanerp(())) ()llga()))) max f(x
(2f=)) V log - (x)
and using& a similar trick as in denoising score matching (Vincent 2011 I Song & Ermon 2019), one
can transform the minimization of Drshhr into the following minimax problem which tractable
min max (p [ = Eg
(x) (S()
= log go (x) ( (
(4)
In practice, f (x) is parametrized using neural networks. The above minimax optimization problem can be efficiently solved by optimizing & and ø alternately.
Hierarchical semi-implicit variational inference
The semi-implicit variational family (x) in equation I is indeed single-layer model in the sense that it contains only one conditional layer. Our main idea is to expand this single-layer semi-implicit variational family into its multi-layer variants and introduce sequence of auxiliary distributions to guide the semi-implicit distributions toward the target distribution. This leads a new SIVI method which we call hierarchical semi-implicit variational inference (HSIVI). We start with the following definition which is motivated by equation (
Definition (Hierarchical Semi- Impliit Distribution). Let Φr qr (x(T)) for some N", where qr (xrr) is called the variational prior. Let qe [z+11 Φ1) be the t-th conditional layer for 0 < < T 1. Denote {Øk) { by Øzt- The t-th layer hierarchical semi-implicit distribution ge (x 22)) is defined recursively from T 1 by
qe (21)))321)) (1 (22+1; de 44+1 (#e41; d2 OSIST-1,
(5)
where qr (WT; &2T) == (xT)). Here, the t-th conditional layer (1 dt) is required to be explicit and reparametrizable with tractable score function Va, log qi (x x++1: Φt).
Compared to the single-laye semi-ing amni-impicii variational family T the family of hierarchical semi- implicit distributions provides principled way to construct more expressive mixing layers using multi- layer architectures. Also, unlike the hierarchical variational models (Ranganath et al. 2016) which require an extra reverse model and explicit variational prior, hierarchical semi-implicit distributions inherit the advantage of SIVI that allows q: (æ;; Φ2+) to be implicit, and as shown next, they do not require a reverse model and can be progressively trained using the simple algorithms of SIVI for each conditional layer, from = T to 0.
3

---

Algorithm 1 Hierarchical semi-implicit variational inference (sequential training)
Input: Auxiliary bridge {pr (x)( po initial value of parameters g0()) { T- (-)-1 Output: The optimal parameters o"
Initialization: < (0)
for = T
to while not converge do Sample a minibatch {2)
(k) 1K from the variational prior qu (xr).
ift t < T - 1 then
Sequentially sample { 1 through g( 01) from i= T 1
(k)
Detach the computation graphs from (2rf4) {x=41/ 1k=1-
end if
Update de by optimizing the Lsivi (1) based on the minibatch (a/ 1 end while
di 4- ds. end
6" + {0r [0r)7 1=11-11
3.1 Progressive approximation with the auxiliary bridge
In this section, we introduce a bridging technique for progressively approximating the target dis- tribution p(æ) using hierarchical semi-implici distributions. Rather than approximating p(æ) with (x( Φ20) directly,
construct sequence of intermediate auxiliary distributions (p(( as a bridge between the target distribution po (=)) := p(æ) and an toap-ro-pppo distribution pr 1 to amortize the difficulty of one-pass fitting. typical example of an auxiliary bridge is the geometric interpolation as described below.
Example (Geometric Interpolation). Lei S(x)) == V log p(x) be the score function of target distribution p(æ:) and Showe (z) = V log Phosel (x) be the score function of base distribution Phose (1)). In geometric interpolation (Neal 2007 Bernton et al.) 2019, each auxiliary distribution pe(ar) for
0 St t s T - 1 has the following probability density function (pdf) and score function
pe( (x) p(x)*' S.(x) =- V log (1 Stare()) + S,)(
(6)
where {A is non-negative decreasing sequence satisfying Ào 1.
Intuitively, we expect the distance between two neighboring distributions (x(x)) and (++1(x) to be not too large so that it would be easy to construct conditional distribution qe (x [ such that (x,, f Qe (xv, ( P (x)+11 ) Note that the auxiliary bridge (p.())) 1 does not necessarily need to have analytical pdfs (up to a constant). In fact, suffices if they have tractable score functions {=))) which lead to another type of auxiliary bridge (Example in Section
3.2 Sequential training of HSIVI
Given the auxiliary distributions (p.(#,) ( hierarchical semi-implicit distribution (Zr; (2)) to match (xx;) from t = T 1 to f = 0. Let the parameters Φ; in the t-th conditional layer be independent across different ts. We first train (1----------1 to match ( by optimizing Φr-1 w.r.t. the single-layer SIVI objective For 7' 2,..., O, given the trained semi-implicit distribution qt+1 [I 12(+1), we can fix it as the mixing layer and train the f-th conditional layer qe (z 1 Φ+) by optimizing de w.r.t. the single-layer SIVI objective LSIVI-/ (pr(x.) (14 (a;; Φ2+)) as well. Note this is fine as the mixing layer can be implicit in SIVI Here, is some distance criterion, e.g. LSIVILB equation (2) or LSIVI.M in equation (4) In this article, we mainly focus on LSIVILA and LSIVI while other distance criteria can also be applied. We summarize this sequential training procedure Algorithm
T-1 - natural approach is progressively train the
LSIVE.y (Pr-1 -
Score based training In addition to the common assumption that (x( is known up to a constant, it is worth noting that LSIVLLBB a is also applicable when only the score functions (z)) T-1 are available which is important for the diffusion bridge construction of auxiliary distributions in Example
Concretely, assume qe (xy (2+11 6.) ) is induced by parametrized transform T; h2 E; 6:)

---

where ~ (e( is random noise. The only term in LSIVILBB (pt (x Φ2()) containing (x,),) is E (mmosddge)) log pr (xx, (see equation whose gradient takes the form
V de E (maste))) log (x:) = Ex (p))))) (11 6i)) V (x( (7)
In the training of HSIVI-SM, each term LSIVI-MM (pr (2;, Il (wvr d2()) involves nested optimization of fe(æ;; tx). When the score functions are computationally expensive, we find that an alternative
parametrization (are; 1/2) == S((w)) - gr ( is useful avoid the time-consuming evaluation of (xz) when optimizing 2h in equation Appendix C.2 for the proof of Proposition !
The reason for this lies in Proposition !
See
Proposition 1. Lel qe (ate, 1 (22)) (0 de) (2re+1; 022+1) ) The minimax optimiza- tion of SM (pe( (a: (21)) is equivalent to
min 6e min Ws
E (2 (m)) g gx(; (S(())) + ge(;;; 2V,,, (o (1-1:
v.) Va, log qi (x), (.)1
Marginal approximation v.s. joint approximation Previous works (Bernton et al. 2019 Bao
al. 2022) often construct joint distribution p(ro:r) and minimize KT(p(20oTT-1 ] g -0:7-1)
where ( is variational distribution. In HSIVI, we directly approximate pr( using the
semi-implicit variational distributions. When p( complex and 7 is small, the variational distribution ((2oTT-1) may be insufficient to fully capture the joint distribution p(æ0:r-1). For example, the optimal fit of the joint distribution for diffusion models established by Analytic- DPM (Bao et al. 2022) does not guarantee that the marginal distributions would be approximated well (see Table? for comparison).
4 Application to diffusion model acceleration
4.1 Review of diffusion models
Recently, diffusion models have shown great success many generative modeling benchmarks, including image generation (Ho al. 2020, Song et al.. 2020a b). graph generation (Niu et al., (020). and text generation (Austin et al. 2021). Diffusion models work by adding noise to the training data in the forward process and then removing the noise to recover the data the backward process, which can be integrated into general stochastic differential equation (SDE) framework. The forward process {ua}sej0.4 is usually described by
du. = f(u., s)ds + g())d
440 ~ ~p(-))
(
where po(+) is the data distribution, w. is a standard Brownian motion, and f(u,, 8) and g(s) are the drift and diffusion coefficients respectively. To generate samples from the data distribution, one can run the following backward process
du, = [f(u,, s) (s) Vu, log (u,))us + g())du, uy ~ PL(-),
where (.(.) is the pdf of u. and w. is a standard Brownian motion when time flows from L to 0. As the score function V., log p. (u.) is intractable, we need to estimate it by denoising score matching (Vincent) 2011: Song et al. 2020b). See more details of diffusion models and the training objectives in Appendix|A
4.2 Diffusion model acceleration via HSIVI
While diffusion models prove effective for generative modeling, often takes large number of discretization steps in the backward process ( to produce high quality samples, which caps their potential for real time applications. Note that the forward process (8) naturally provides another type of auxiliary bridge, which combined with HSIVI, can be used to accelerate the sampling process of diffusion models.
Example (Diffusion Bridge). Consider the forward process (ux)es(o,1) with 0 (defined in equation 8) in diffusion models. We choose T discrete time steps 0 z so < < ST-1 S. and
5

---

let a; := Us with probability density function Pt (-). Assume each auxiliarr distributions Pc(-) for
s t S admits a score function as
(z)x)= == V po))(()) R (x, 8t), 0 0S1ST-1,
where S (x) is a pre-trained score model with the denoising score matching loss (equation (3) in Appendix(A] Let us denote S (x, st) by St (z) for short. With sufficient samples from the data distribution (x) and model capacity, the approximation (z)x) can be reasonably accurate for almost all x and t (Song et al. 2020b).
As the pre-trained score model provides a diffusion bridge from the simple distribution PT-1 (e.g.. standard Gaussian) to the data distribution, we can train the hierarchical semi-implicit distributions to approximate the diffusion bridge within the HSIVI framework. Given the expressiveness of hierar- chical semi-implicit distributions, we may expect an accurate approximation of the data distribution with small number T and hence acceleration can be achieved.
However, the memory usage during the sequential training process for HSIVI might be large because of the necessity for independent parameters. Therefore, we may employ parameter sharing scheme
which is commonly assumed in diffusion models (Song & Ermon 2019: Ho al. (2020) such that
different conditional layers share the same parameters Φ. Note that sequential training is not suitable in this setting. Therefore, we propose a joint training procedure that minimizes a weighted sum of the SIVI objectives
L (0) = Σ C (p((m))lla
(10)
where (()) : (O...., 1 -> RR,, is a positive weighting function and f is some distance criterion. See Algorithm 2 in Appendix C.3 for more details of joint training.
More specifically, this work, we mainly focus on building the diffusion bridge with variance preserving SDE (VP-SDE) (Song et al. 2020b) such that u [4g ~ N (VN(aa(()uu)))) (1 11--(1 (I)) with decreasing function ass) of We use LHSIVI-SM in equation (10) for training and set the weighting function (t) = 1 - (xx)) as recommended in Song et al. (020b), which tends to train layers that far from t = first during the training, resembling the sequential training. Another popular
formulation of diffusion models is to fit a noise model c" (z, s) that predicts the noise added to a noisy sample at time (Ho et al. 22220) HSIVI-SM also generalizes to the case where a pre- traineained noise model is available. The pre-trained noise model forms a (generalized) diffusion bridge by letting e (a) e" (ax, 8t), and we call the corresponding training method "e-training". We provide reparametrized objective function CHSIVISM for e-training in Appendix C.4
Several efforts have been made to accelerate the sampling process of diffusion models, including
faster numerical ordinary differential equation (ODE) solvers (Song et al. (2020a] [Zhang & Chen
2022; Lu et al. (022] and distillation techniques (Luhman & Luhman [202]] Salimans & Ho [022]
Zheng et al. (0022). Our approach is different from these previous efforts in that we accelerate the
stochastic diffusion model directly (hence would provide more diverse samples (Figure (6) and do
require sampling datasets from the diffusion models prior to distillation which is computationally expensive. From a Bayesian perspective, HSIVI is related to Song & Ermon (2019). where the
authors used the annealed Langevin dynamics guided by pre-trained score model to sample from the data distribution. By solving this problem using variational inference approach, HSIVI enjoys faster sampling speed and scales better high-dimensional data.
Experiments
In this section, we first compare HSIVI to its single-layer counterpart, SIVI, on two inference tasks. We use the sequential training method where each conditional layer in the hierarchical semi- implicit variational distributions has independent parameters. We then apply HSIVI-SM to diffusion model acceleration on various datasets. As the memory consumption for generative models is large, we use the joint training method where the conditional layers in hierarchical remi-implicit distributions have shared parameters across different (s. For all experiments, each conditional layer is modeled as Gaussian distribution with parametrized mean and variance. More details of the model architectures and hyper-parameters are included in Appendix The code is available at https:/ / thub. www./onginY///HIII....

---

Ground Truch
SIVI-LB
HSIV-LB (T=5)
SIVI-SM
HSIVISMM (T=5)
15
-10
35
-10
-2
Figure 2: Comparison of 10,000 generated samples from SIVI and 5-layer HSIVI on two- dimensional Gaussian mixture model (blue).
Ground Truth
SIV-SSM
HSIVI-SM (T-5)
0.2
95% Confidence Intervol
0.2
95% Confidence
0.2
25% Cant dence Interva
0.2-
Observations
Opservations Semsle Mean
observations
Mean
0.4-
3.4
D4-
-0.8
-2.
-0.6
-0.8
-0.-
-1.0-
-1.0
-1.--
-1.2
-1.2
0.0
0.5
1.0
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
155
2.0
2.5
3.0
Figure 3: The posterior estimates obtained by different methods. For each method, collect 100,000 samples to calculate the sample mean and confidence interval.
5.1 Target distribution approximation
Gaussian mixture model We first evaluate HSIVI and SIVI two-dimensional Gaussian
mixture model. The target distribution p(æ) takes the form p(ar) = Σi= 1/8 Nax;; p4; o²I) where
μ4 = [10 cos( 1zt 10 sin( 1x )²" a 1. For HSIVI, we construct an auxiliary bridge of T = 5 with geometric interpolation in Example [ where (zase(x) N(; 0, I) and X, 1 t/5. The results are presented in Figure 2 Note that the modes in this Gaussian mixture model are far apart from each other, and both SIVI-LB and SIVI-SM are trapped in local modes. In contrast, both HSIVI-LB and HSIVI-SM discover all modes and provide an accurate approximation of the target distribution with HSIVI-SM being better for recovering the right scale of variance.
4
High-dimensional conditioned diffusion The second example is a high-dimensional Bayesian inference problem arising from the following Langevin SDE
dæ 10x,(1 (1- ) dw..
(11)
where To = O and wa is me-dimensiional standard Brownian motion. This system describes the motion of a particle with negligible mass trapped in an energy potential with thermal fluctuations represented by the Brownian forcing (Cui et al. 2016). Using Euler-Maruyama scheme with step size As 0.01 on a time interval [0, 3), we discretize the SDE (TT) into æ (Zd,..... Id where d, 0.01i, which gives the prior distribution Ppeior (x) of the 300-dimensional variable The noisy observations is obtained by y x + E. where & ~ (N, the posterior distribution of the latent states p(xly) oc Pprior (x) p(()zz). The ground truth is formed by running 100,000 independent stochastic gradient Langevin dynamics (SGLD) chains with step size of 0.0001 and collecting the results after 10,000 iterations.
with 0.1. Our goal is to infer
For HSIVI, we form the auxiliary bridge using geometric interpolation with Pa)se()) = Nxx y, I and X;
t/ 1) for O..... 1. Figure 3 shows the estimated posteriors obtained by different methods. We see that SIVI-SM severely underestimates the variance. With T = 5 layers, HSIVI-SM fits the variance better and hence provides more accurate posterior estimates. For both HSIVI-SM and HSIVI-LB, the estimated covariance matrix becomes more accurate as T increases (Table 4 in Appendix D.2). demonstrating the effectiveness of hierarchical models for fitting complicated distributions.
7

---

2
Sample trajectories generated from 10-layer HSIVI-SM on four 2D toy examples. The arrows represent the estimated score function in HSIVI-SM. The sample size is 10,000.
Figure 4:
Table 1: JS divergences between the target distribution and the variational approximation on the four toy datasets. The results of HSIVI-SM are averaged by independent runs with standard deviation in the subseripts. JS divergences calculated by the ITE package (Szabó (014) with 10,000 samples.
T
T 10
T 1000
Name
DDPM
DDIM
HSIVI-SM
DDPM
DDIM
HSIVI-SM
DDPM
Checkerboard
0.891
0.591
L0.006
0.521
0.373
0.030
0.058
Swissroll
1.037
0.332
0.126
0.334
0.164
0.082
0.042
Circles
0.907
0.397
0.003
0.364
0.201
0.073
0.032
Moons
0.961
0.355
0.096+00013
0.352
0.137
0.059
0.036
5.2 Diffusion model acceleration
2D toy examples In this toy model example, we test four synthetic 2D datasets: Checkerboard, Circles, Moons, and Swissroll (Pedregosa et al. 2011). We first pre-train the score model S" (ar, 8)
for s E [0, 1] with quadratic noise schedule a(s) s² For constructing the T T-layer diffusion bridge, we select {sr] 12'-1 so that a(s)) = [0.01 + (Vo.8 0.01)t/T]². Figure shows the sample trajectories (Iy, Ø7, 2s and xx) progressively generated from 10-layer HSIVI-SM. We see clearly how the semi-implicit distributions are guided towards the target distribution and all modes discovered. We also report the AnsennShannoo (JS) divergence between the target distributions and the estimated distributions in Table We see that HSIVI-SM significantly improves upon DDIM and DDPM both cases with and steps. Also, 10-layer HSIVI-SM comparable to DDPM with 1000 full steps. See Figure 10 in Appendix D.3 for visualization of samples from different methods.
the noise model 6" 6"(xx, instead of the score model and e-training to train HSIVI-SM. The structure of <"(x,, 8) follows the UNet in Ho et al. (2020) by reducing the number of input and output channels one. With the same noise schedule employed in [Song et al. (020a), we first pre-train the noise model E" e"(x, s) with 1000 discretization steps and then form the Tllayrr diffusion bridge for HSIVI-SM by selecting discrete time steps. Figure shows the samples from DDPM, DDIM, and HSIVI-SM with T = 5 steps. We see that the samples produced by HSIVI-SM are much cleaner and more recognizable than those produced by DDPM and DDIM.
MNIST On MNIST,
CIFAR-10, CelebA & ImageNet On both CIFAR- and CelebA, the structure of pre-trained noise model e" (x, s) follows the UNet structure structure/Romnebergaa et al. 2015) employed by Ho et al. (2020), instead of the huge VP deep continuous- model (Song al. 2020b) that has more channels and layers. We also provide additional results on ImageNet (64 664x64) with more powerful pre-trained score nets in (Nichol & Dhariwal (bigger models with more parameters). Since this generative modeling has been formulated as a score-based VI problem, we do not have to use any training data for training HSIVI-SM.
Following the noise schedule employed in Song et al. (2020a), we first pre-train the noise model c" s) with 1000 discretization steps and then form the 7 T-layer diffusion bridge for HSIVI-SM

---

DDPM (T-5)
(T-5)
HSIVI-MM (T-5)
6
9
7
36
6
9
2
7
7
/
5
7
6
9
7
7
4
6
3
8
7
1
9
7
7
6
2
9
7
3
O
8
4
3
8
5
D
6
6
2 3
5
3 3
0
0
6
7
8
3
2
3
8
6 8 8 4
7
7
2
4
9
3
8
5
6
2
4
8
8 8
5
3
2
3
2
6 8
3
0
3
7
0
3
6
2
8
3
6
3
6
4
9
9
9
Figure 5: Comparison of the quality of uncurated samples generated by DDPM, DDIM, and HSIVI
SM with 5 discrete time steps on MNIST.
Table Sample quality measured by FID (4) on CIFAR-10, 10, CelebA and ImageNet with a varying number of function evaluations (NFE). Results of baselines are calculated by running their official codes, where the architectures of score model (or noise model) are the UNet employed in Ho et al.
(2020) for CIFAR-10 and CelebA and (Nichol & Dhariwal 2021) in ImageNet.
Dataset
CIFAR-I0 (32x32)
CelebA (64x64)
ImageNet (64x64)
NFE
5
10
10
15
5
10
15
DDPM et al
2020
320.16
278.65
198.00
366.10
309.95
206.92
402.68
358.80
284.00
DDIM (Sngg el M.
2020n
41.53
13.73
8.78
27.38
10.89
7.78
147.03
42.31
24.85
FastDPM KKng Pine (2021
67.64
9.85
6.16
27.63
15.44
12.05
N/A
N/A
N/A
Mualytic-DDPM [iso el at.
2022
93.16
34.54
20.03
50.92
28.93
21.84
N/A
60.65
45.98
Anulytic-DDIM [Bao el al. 2022
51.86
14.08
8.65
29.40
15.74
12.25
N/A
70.62
41.56
DPM-Solverrfast
329.13
10.89
4.67
355.96
402.43
20.03
HSIVI-SM (ours)
6.27
4.31
4.17
6.22
3.09
2.23
40.43
17.67
15.49
by selecting T discrete time steps as before. For HSIVI-SM with e-training, the conditional layer ge pe(-Ja [ Φ) is modeled as a Gaussian distribution with mean 14: (1x 6*) and diagonal variance matrix Ee(d") where {Øt ) Φ are the variational parameters. In our implementations, both Ue (2:+1 )²) and f:(r;; v)) use the same architecture as E c" s). The number of layers, which is also the number of function evaluations (NFE), is set to be T = 5. 10, 15 in our experiments. We train HSIVI-SM with the same setting for 10, 15. The 5-layer HSIVI-SM is trained by further fine-tuning the well-trained 15-lryer HSIVI-SM and we find this strategy leads to better results. During each nested training loop of ft ( v we update v 20 times before each update of o, since find (z, ()) needs more training empirically to provide reliable guidance.
For each method, we draw 50,000 samples and use the Fréchet inception distance (FID) score (Karras et al. 2022) to evaluate the sample qual- ity (Table 2). We find that HSIVI-SM performs on par or better than the other baselines on both
seed
seed
Seed 3
Seed
Seed
CIFAR-10 and CelebA, and the advantage is evident when the NFE is small. The sampling trajectories of 10-layer HSIVI-SM with the same starting point but different ran- dom seeds are shown in Figure 6 We that HSIVI-SM is capable of producing more diverse samples due to its stochastic nature, which is dif- ferent from existing ODE based fast diffusion model samplers.
CelebA
Figure 6: Sample trajectories of 10-layer HSIVI- SM with the same starting point 210 on CelebA.
5.3 Additional Study
Ablation of layers number In Figure We provide failure case fitting the checkerboard target with diffusion bridge, demonstrating that the HSIVI-SM algorithm fails when the layer number 7' is small (the distances of auxiliary distributions at successive time steps are large) on checkerboard

---

T=1
T=2 T=3 T=4 T=5
Figure 7: Failure cases of HSIVI-SM. The quivers show the estimated score by the f function. T is the layers number of HSIVI-SM. The generated samples in orange show that smaller 7 may fail this example.
Table 3: Comparison of non-isotropic conditional layers and isotropic conditional layers on CIFARIO, the sample quality is measured by FID ()).
NFE
DDPM
DDIM
HSIVI-SM (isotropic)
HSIVI-SM (non-isotropic)
5
320.16
41.53
7.33
6.27
10
278.65
13.73
4.78
4.31
15
198.00
8.78
4.46
4.17
distribution. In fact, the score function on the checkerboard target is sharp on the boundaries but vanishes elsewhere. Therefore, fitting this target distribution is somewhat challenging.
Ablation of the variational family To validate the improvement of HSIVI-SM on diffusion models, we train HSIVI-SM with isotropic conditional layers in consistency with denoising-diffuss sampling, like DDPM and DDIM. We report the results of FID on the CIFAR-I0 dataset Table B These results provide further evidence for the statement outlined in Section 3.2 HSIVI-SM matches the marginal distributions ge (2x,) and pt (x;) directly via score matching and would ensure better fit for Po (20). The enhancement of HSIVI-SM over DDPM stems not only from its more expressive variational distribution but also from the direct alignment of the marginal distributions.
6 Conclusions
We introduced HSIVI, hierarchical semi-implicit variational inference method that enables more expressive multi-layer construction of semi-implicit distributions. Given appropriate auxiliary distri- butions that interpolate between simple base distribution and the target distribution, the conditional layers in hierarchical semi-implicit distributions can be progressively trained one layer after another. In experiments, we showed that HSIVI outperforms previous single-layer SIVI methods on several Bayesian inference tasks with complicated posteriors. HSIVI can also be used to accelerate the sampling process of diffusion models, where pre-trained score networks serve natural sequence of bridging distributions, which allows for direct acceleration of the stochastic diffusion model and does not require expensive sampling from the diffusion models during training. We showed that HSIVI can produce high quality samples comparable to or better than existing fast diffusion model samplers with few function evaluations on various datasets. Limitations are discussed in Appendix F
Acknowledgements
This work was supported by National Natural Science Foundation China (grant no. 12201014 and grant no. 12292983). The research of Cheng Zhang was supported in part by National Engineering Laboratory for Big Data Analysis and Applications, the Key Laboratory of Mathematics and Its Applications (LMAM) and the Key Laboratory of Mathematical Economics and Quantitative Finance (LMEQF) of Peking University. The authors appreciate the anonymous NeurIPS reviewers for their constructive feedback.
10

---

References
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 14:11791-17999 2021.
Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dmm: an analytic estimate of the optimal reverse variance diffusion probabilistic models. arXiv preprint arXiv:2201.0650: 2022.
Espen Bernton, Jeremy Heng, Arnaud Doucet, and Pierre E. Jacob. Schrödinger bridge samplers. arXiv preprint arXiv:1912.13177 2019.
Christopher M. Bishop and Michael E Tipping. Variational relevance vector machines. In Proceedings of the Sixteenth conference Uncertainty in artificial intelligence, pp. 46-53, 2000.
David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: A review for statisticians.
Journal of the American Statistical Association, 1 112:859 - 877, 2016.
Tiangang Cui, Kody JH Law, and Youssef M Marzouk. Dimension-independee likelihood-informec
memc. Journal of Computational Physics, 304:109-137 2016.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. In International Conference on Learning Representations, 2017.
Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Score-based generative modeling with critically- damped langevin diffusion. In International Conference on Learning Representations (ICLR), 2022.
R. J. Giordano, T. Broderick, and M. I. Jordan. Linear response methods for accurate covariance estimates from mean field variational bayes. In Advances in Neural Information Processing Systems, 2015.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Tianyang Hu, Zixiang Chen, Hanxi Sun, Jincheng Bai, Mao Ye, and Guang Cheng. Stein neural sampler. arXiv preprint arXiv:1810.03544 2018.
Ferene Huszár. Variational inference using implicit distributions. arXiv preprint arXiv: 1702.08235. 2017.
Aapo Hyvärinen. Estimation of non non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4):695-709, 2005. URL hhtp:://jmlr http://jmllr.org/ppprrss. org/papers/v6/ hyvar hyvarinen0ba. html
html
T. S. Jaakkola and M. I. Jordan. Improving the mean field approximation via the of mixture distributions. In Learning in Graphical Models, pp. 173-173, 1998.
Michael I. Jordan, Zoubin Ghahramani, T. Jaakkola, and Lawrence K. Saul. An introduction to variational methods for graphical models. Muchine Leurning, 37:183-233, 1999.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Proc. NeurIPS, 2022.
D. Kingma and J. Ba. Adam: A method for stochastic optimization. ICLR, 2015.
D. P. Kingma and M. Welling, Auto-encoding variational bayes. In ICLR, 2014.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models, Advances in neural information processing systems, 34:21696-21707, 2021.
Diederik P. Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems, pp. 4743-448 4751, 2016.

---

Zhifeng Kong and Wei Ping. On fast sampling of diffusion probabilistic models. arXiv preprint wrXiv:216600013. 2021.
Qiang Liu, Jason Lee, and Michael Jordan. A kernelized stein discrepancy for goodness-of-th tests. In International conference machine learning, pp. 276-284. PMLR, 2016.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. arXiv preprint arXiv:2206. 00927, 2022.
Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint 1iXi::2001.02888, 2021.
L. M. Mescheder, S. Nowozin, and A. Geiger. Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning, ICML. 2017, Sydney, NSW, Australia, 6-11 August 2017, 2017.
V. Moens, H. Ren, A. Maraval, R. Tutunov, J. Wang, and H. Ammar. Efficient semi-implicit variational inference. arXiv preprint arXiv: 100.v0200..0607, 2021.
Radford Neal. MCMC using hamiltonian dynamics. In S Brooks, Gelman, G Jones, and XL Meng (eds.), Handbook of Markov Chain Monte Carlo, Chapman & Hall/CRC Handbooks of Modern Statistical Methods. Taylor Francis, 2011. ISBN 9781420079425. URL http://books
Radford M Neal. Annealed importance sampling, Statistics and computing, 1 11:125-139, 2001.
Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. 2021.
Chenhao Niu, Yang Song, Jiaming Song, Shengjia Zhao, Aditya Grover, and Stefano Ermon. Permu- tation invariant graph generation via score- based generative modeling. International Conference on Artificial Intelligence and Statistics, 2020.
J. J. Nott, S. L. Tan, M. Villani, and R. Kohn. Regression density estimation with variational methods and stochastic approximation. Journal of Computational and Graphical Statistics, 21(3)7977-800, 2012.
J. W. Paisley, D. M. Blei, and M. I. Jordan. Variational bayesian inference with stochastic search. In Proceedings of the 29th International Conference on Machine Learning ICML, 2012.
G. Papamakarios, E. Nalisnick, D. Rezende, S. Mohamed, and B. Lakshminarayaman. Normalizing flows for probabilistic modeling and inference. ArXiv Preprint arXiv: 2019.
Fabian Pedregosa, Gaël Varoquaux. Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-leam: Machine learning in python. the Journal of machine Learning research, 12:2825-2830, 2011.
R. Ranganath, S. Gerrish, and D. M. Blei. Black box variational inference. In AISTATS, pp. 814-822. 2014.
Rajesh Ranganath, Dustin Tran, and David Blei. Hierarchical variational models. In International conference on machine learning. pp. 324-333. PMLR, 2016.
D. Rezende and S. Mohamed. Variational inference with normalizing flows. In Proceedings of The 32nd International Conference on Machine Learning, pp. 1530-1538, 2015.
D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. International Conference on Machine Learning, 2014.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computeras- sisted intervention. Springer, 2015.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. ArXiv, abs/2202.00512, 12, 2022.
12

---

L. K. Saul and M. I. Jordan. Exploiting tractable substructures in intractable networks. In Advances in Neural Information Processing Systems, 1996.
J. Shi, S. Sun, and J. Zhu. Kernel implicit variational inference. In International Conference on Learning Representations, 2018a.
J. Shi, S. Sun, and J. Zhu. A spectral approach to gradient estimation for implicit distributions. In International Conference Machine Learning, 2018b.
Artem Sobolev and Dmitry P. Vetrov. Importance weighted hierarchical variational inference. In Neural Information Processing Systems, 2019.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020a.
Y. Song, S. Garg, J. Shi, and Ermon. Sliced score matching: scalable approach to density and score estimation. In Proceedings of the Thirty-Fifth Conference Uncertainty in Artificial Intelligence, 2019.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 2020b.
Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density ratio estimation in machine learning. Cambridge University Press, 2012.
Zoltán Szabó. Information theoretical estimators toolbox. Journal of Machine Learning Research, 15:283-287, 2014.
Michalis K. Titsias and Francisco J. R. Ruiz. Unbiased implicit variational inference. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 167-176. PMLR, 2019.
D. Tran, D. M. Blei, and E. M. Airoldi. Copula variational inference. In Advances in Neural Information Processing Systems, 2015.
D. Tran, R. Ranganath, and D. M. Blei. Hierarchical implicit models and likelihood-free variational inference. In Advances Neural Information Processing Systems, 2017.
Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. In Neural Information Processing Systems (NeurIPS), 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computa- tion, 23(7):1661-167, 2011. doi: 10.1162/NECO, a a_00142.
J. Wainwright and M. I. Jordan. Graphical models, exponential families, and variational inference. Foundations and Trends in Maching Learning, (11-2):1-305 2008
Mingzhang Yin and Mingyuan Zhou. Semi- Semi-implicit variational inference. In International Conference on Machine Learning, pp. 5646-5655, 2018.
Longlin Yu and Cheng Zhang. Semi-implich variational inference via score matching. In The Eleventh International Conference on Learning Representations, 2023. URL https://openrevieer net/ forum?id=ssin Sarumiid=sd9022ytrrt
C. Zhang, Shahbaba, and H. Zhao. Variational hamiltonian monte carlo via score matching. Bayesian Analysis, 13(2):485-51) 506, 2018.
13

---

Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. In NeurlPS 2022 Workshop on Score-Based Methods, 2022.
Hongkai Zheng, Weili Nie, Arash Vahdat, Kamyar Azizzadenesheli, and Anima Anandkumar. Fast sampling of diffusion models via operator learning. In NeurIPS 2022 Workshop on Score-Based Methods, 2022.
14

---

A Details of diffusion models
Diffusion models work by adding noise the training data in the forward process and then removing the noise to recover the data in the backward process, which can be integrated into general stochastic differential equation (SDE) framework (Song et al. 2020b). The forward process {u, (e())) is
usually described by the SDE
du. f(us, s)ds g(s)dws, Un ~p(())
where (-) is the data distribution, w, is a standard Brownian motion, f(us, 8) and g(s) are the drift and diffusion coefficient respectively. To generate samples from the data distribution, can run the following reversed SDE
du, [f - V.) logps(u,))d= + g(s)dw,, uL "L(P)()
where (-(-) is the probability density function (pdf) of u. and w. is a standard Brownian motion when time flows from L to There exists deterministic process shares the same marginal probability densities (p.(-) described by the following ordinary differential equation (ODE)
1 du = [f( s) (g log (og))
UL ~ (-L()),
called probability flow (PF) ODE.
In practice, Song et al. (2020b and Kingma et al. (2021) designed several examples of the forward
process such that it diffuses the data distribution (o(.) to fixed unstructured distribution pL (-).
Here we mainly consider the Variance Preserving SDE (VP-SDE) used in DDPM (Ho et al. 2020
Song et al. 2020b), Let the drift coefficient f(u,, s) = dloga(s) and the diffusion coefficient
2ds
(s(s) dlogols)
where a(s) E I' is a decreasing smooth function with (0) = l, ~ 0.
de
Then the distribution of u, conditioned ug is explicit as
Uu ~ N aa(s) (1 a(s)))) i.c. u, = Va(s)u + vI a(s),,
(12)
where e a standard Gaussian noise. In practice, diffusion models use a neural network So u.. s) to approximate the score function So(u,, s) by optimizing the denoising score matching objective (Vincent 2011)
2 / (ls))
Lasa (0, (x))
So (u, 8) e/v1 a(s) ds.
(13)
where (ss) s) is positive weighting function. Instead of modeling the score function, Ho 2020)
proposed to predict the conditional noise based on u, This leads to the following DDPM loss
Ladpm (0, (s)) 1 I/" ((s) (us))Burrpodueeee- (u.(u.. s) - =l
(14)
where (s) is a positive weighting function. fact, we have the relationship
So 8) = -Eo(u, s)/ / a(s).
(15)
We call Lasm ixcore-predictionn training and faapm "c-prediction" training.
With the pre-trained score model (s,,,,) or noise model ég (u(u,,)), s), Song et al. (020b) shows that the samples of (-(-) can be generated by simulating the backward SDE, e.g. the sampling scheme of DDPM (Ho et al. 2020). Moreover, Bao et al. (2022) proposed Analytic- discretization form responding to the KL divergence of the joint distribution on the discrete time
the optimal
steps. Also, several high-order ODE solvers (Song et al., 2020a; Zhang & Chen 2022
Lu et al.
2022 were proposed achieve faster sampling.
B
More details of UIVI
Unlike optimizing the surrogate ELBO, Titsias & Ruiz (2019) proposed unbiased implicit variational inference (UIVI) which relies unbiased gradient estimator for the exact ELBO. To elaborate further, reparametrize the conditional 4o (2/x) such as x = To (z, E), ~ (E))), then
ELBO = V (e
(Dgg(D,zu) xx) ) log qo (
E (e)
E) ghth (z,<)) [
mod
15

---

where
I 2, 6) = V log D, x)) (z.22) VoTo(z,E),
mod
em 2 E) = (g)(=((a) (m) V log do 4o (2/|2)) Tx.xe) Volo(z,e).
(16)
The second gradient term Dg involves an expectation w.r.f. the reverse conditional (= which is estimated by an MCMC sampler in UIVI. However, the inner-loop MCMC runs may require long iterations for convergence.
C More details of HSIVI
C.1 Score-based training of HSIVI-LB
In the sequential training of HSIVI-LB, although the objective LSIVLLB (px(x:) [age Φ2+)) is calculated based on (ar),, the gradient of w.r.t. Φe has a closed form containing only the score function (x) without knowing the corresponding pdfs. This derivation is important in the tasks where score functions of the auxiliary distributions are tractable while pdfs (up to a constant) of them are unavailable (for example, the diffusion bridge in Example 2 Concretely, assume the I-th conditional layer Qi [#111 Φb) is induced by parametrized transform 2; = ta (2 Φ)) where (e( a random noise, since qe (2 Φt) reparametrizable according Definition [. The only term in LSIV-LBB (pr(x;) (xx; @21)) containing pe (xx)) is E (mm)t)))) log pa (xx) (see equation (2)) whose gradient takes the form
R ((g(eed))))) log log,t((n)) (xxe) = Vo,
log (h, (;
= (o
(hr(0111 6:)) V hi σ1)
by the chain rule, where Vo, (3 Φe) is the jacobian matrix of h4 (2 e; Φ1).
In our implementation of HSIVI (in both sequential training and joint training), we generally assume the conditional layer qa (= sx) is induced by
h (1 E; ox) = pa (1 Φ.) + D. (at+1: 6t)
(17)
where (x Φe) is positive definite covariance matrix and (NO, I) is standard multivariate gaussian variable. In equation (7). di should replaced by the joint training case.
C.2 Proof of Proposition I
Let (o R++1 Φ21) ) = (1(1//1411 (0 The minimax optimiza- tion of CSIVISMM (pe (2:()))llg)(e) (2()) equivalent to
Proposition I
min
(g(e (2xe) v/)]" (x:)) + (a 2V., m,
min Eqi 1222))
1d2 Il (wi v/r) Va, log q (2:41:
Proof of Propsition 1
The minimax optimization problem of LSIVISMM (po(2:))l )2()) is
min max B (z (4)
Va, log gxggee (2 dc)] Ilf vx)
Φe Ve
according to equation
4 For minimization w.r.t. Φ., this target equivalent to
(m 2 (h) - Va. log qo #1+1i oc)] (z:
Em fo (x, 4h) (x;)
(2 - fe(x,; wh) 2V., log qi (z; Fz++1 Øz) -E [S. 9. (x )² (wx, + (xri v.) 2V.. log q ( dx)]
For maximization w.r.t. vh., this target is equivalent to
By (m), ai (11 (are; obe (S. Va, (2 ox)) (x;
vx) (xx V.. l (2) oi) C
= Eg (ae m ( (2; Va, log qe (x. 2111:
16

---

where C' is term that does not contain th, Therefore, the minimax optimization problem is equivalent
min
(u (z)) (o( (xx + ge 2V (2, ( (b)) Væ, ( 0:)
6c
min
Ve
C.3 Joint training of HSIVI
As mentioned in Section when parameter sharing scheme is used the conditional layers for application diffusion model acceleration, sequential training from T to 0s not feasible. Therefore, we consider the following training objective
T-1
L
Σ B(1) (m)())))lu((m
An intuitive method is to randomly sample batch of time steps {tx [t*] and for each ty train LSIVI-/ (Ptz (zt, ) 4 (x)z;:) directly. However, sequentially sampling It, through g( ( o) from i= - to = fę is still necessary in this case, making it memory-consuming to preserve the computation graphs of the entire sampling process.
In order to reduce the cost of accumulating computation graphs, for each f, we treat ( d) as a fixed mixing layer denoted by 4c+1 (#++1) and only fit the conditional layer dr (01 Φ). More specifically, for HSIVI-SM, we consider the following optimization problem
min
Eq (x:)) g:( x0)²" (S,(x;) + ge(an; ()) - 2V. log (mg)(()(()))))))) (18)
min
(g: Va log (x)
(19)
where (#r, 0e+ ge (wx 1 i Q) (#++1) 1). what follows, we demonstrate that the above problem also ensures an accurate approximation of the target score function.
For equation 19 by the denoising score matching trick (Hyvärinen, 2005), the optimal point of vb, denoted by zi (T),, satisfies
g ( Vx, log (zr;
where ge (ax;;)) ④) g(a,, [2:+11; () 1d)=++) By plugging in the optimal point vi (Ø),, each
term equation (18) equivalent to
Ea (mx)) (i / (+)) (5.(2.) ge " (b)) 2v log (me
= =Ea(( [$] (mx) gi ( (o)) 2 // ((me+1) (5:(m)) - (=( a (o)) V (ma (m 1dd
= Egg(mees))
S (xxe)
g (mx; vi (+)) - / (ax())) g( 10 (+)) Vz, (e dx
(wx)) 2S, (x:)) Vm, log q. (wx; Φ) + (Va, log क (wi }))
-E Es((ero (axe) V., log qe (m:e; 2
Therefore, the global optimal point o² also ensures that the score of the variational distribution fits the target score function.
Based on the training objectives 18 19 mentioned above, we propose Algorithm for joint training,
which does not need to store computation graphs of the sample sequences. Moreover, by assuming an increasing weighting function ((),, we assign larger weights ((1) for those / close to T which tends train the conditional layers that are close sequential training.
first during the training, resembling the
17

---

Algorithm 2 Hierarchical semi-implicit variational inference (joint training)
Input:
Auxiliary bridge (pi(x) HT ; weighting function B(t); initial value of parameters /5 Output:
The optimal parameters s² Initialization: Φ < (b().
while converge do Uniformly sample K time steps (tx kk=o with replacement from (0,...,T-11
Sample minibatch {a.7 ( Jk=1 from the base distribution (r()).
for k = 1,..., K and fg <. - do
Sequentially sample +++++1 through q(ar; [1:41: the) from i 10
Detach the computation graphs from end for
Update by optimizing objective Σt (1e) (pt, (co Alee ( 6)), where k-th term computed based on single sample x: (k) end while o" Φ.
+1+
C.4 c-training of HSIVI-SM
Another popular formulation of diffusion models is modeling the conditional noise c,(Ug, s) by optimizing the DDPM loss in equation (4 where u, Va(s)uu + v1 a(s)E, introduced as "e-prediction" in Appendix A Now, let us assume the diffusion bridge is constructed with VP-SDE and we have a pre-trained model of conditional noise E" (u, s). Similarly, we construct a sequence of noise models (=x) T- by letting 2; = ue and (x) e, (æ, S;) which forms a (generalized) T-layer diffusion bridge. We only discuss how e-training can be applied joint training and the derivation for sequential training is similar. In what follows, we consider the transformation of the joint training objective SUSIVISM for diffusion model acceleration.
By letting the weighting function B(t) where Oe is replaced by o, the objective of HSIVI-SM takes the form
(st) and considering the reparametrization form 17
LuSIvIsM(ó, 15)
Σ T 1..1)))
2 ( ((t)) 1/2
(
(
-Il (lv (2)) 1
(20)
where ST (x;) is a pre-trained score model. Note that we have (z())S((z)) = - (x:) by equation Define
15).
f((a;)) - ((tt)f( (T(41;00) ()(
The HSIVI-SM objective 20) then takes the form
LuSIvIsM( xb)
Σ T (a
(2f [ 1/2 (x++1:)))) (()+)((0))) (m ]|]] (21)
and we call it the objective for e-training. In our implementation of e-training, we directly parametrize fa v and E. ( σ) instead of fa v) and Σ, (x 6). The objective (21) is more numer-
ically stable since the magnitude of (x o) is generally larger than Σ, (z)+1:0)
C.5 Complexity comparison of HSIVI
For methods that we discussed in SIVI variants, which use single conditional layer (i.e., T=1) and hence would be much cheaper to sample from than HSIVI-SM that uses multiple layers T' > 1. For methods that we discussed for diffusion models, the computational complexity would be similar they had the same T. That is because we used the same neural network architecture for the conditional layers in HSIVI and the score nets diffusion models. We have comparison of the sampling time of different methods in Figure 13
18

---

HSIVILBB (t=4)
HSIVI-LB (t-3)
HSIVILBB (t=2)
HSIVI-LB (t=1)
HSIVI-LB (t=0)
28
15
-10
-00
-10
-10
-10
HSIVI-SM (t=4)
HSMI-SM (t=3)
HSIVI-SM (t=2)
HSIVI-SM (t=1)
HSIVI-SA SM (t=0)
-3
-
-20
-15
-3
10
-10
30
-10
-30
Figure 8: Upper row: Sample trajectories progressively generated by 5-layer HSIVI-LB guided by diffusion bridge. Bottom row: Sample trajectories progressively generated by 5-layer HSIVI-SM guided by diffusion bridge.
Ground Truth
HSIVI-LB (T=5)
0.2
95% Confidence Interval
5.2
Confi
0.2
Confidenc
0.3-
0.4
0.4
004
-0.6-
--66
0066
-15-
-.0
-.0-
1.2-
0.0
0.5
1.0
2.0
2.5
3.0
0.0
0.5
3.0
3.55
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Figure 9: The posterior estimates for conditioned diffusion obtained by SIVI-LB and 5-layer HSIVI LB. For each method, we collect 100,000 samples to calculate the sample mean and confidence interval.
D Additional results of experiments
D.1 Gaussian mixture model
For HSIVI on the Gaussian mixture model, the auxiliary distributions can also be constructed with diffusion bridge in Example Concretely, the diffusion bridge constructed by
D | (Vaarao (1-a, I).
To Po(z)).
where = a(s)) with as)) defined in equation 112 In this example, the score function S,(x:) = Vz, (xx:) has an analytical form
Sa(..) = So (wi Voru, (aro² +1 - + 031ST-1.
where So(æ; μa, I²I) is the function of the Gaussian mixture model p(æ; pt, = 1/8 N(; μ4 I)). We set the number of layers T and 1 t/5 for t O,..., Figure shows the sample trajectories generated by HSIVL. We see clearly that semi-implicit distributions are guided toward the target distribution following the diffusion bridge.
D.2 High-dimensional conditioned diffusion
We also test SIVI-LB and HSIVI-LB for fitting the posterior in high-dimensional conditioned diffusion. The auxiliary bridge is formed using the same geometric interpolation as for HSIVI-SM,
19

---

DCIN
COM
Oreund Trute
DORM (T-1)
HISW-SH (T+5)
Ground Truhh
COMNTT-30
DONN (T-55
HSRASS (T-5)
(a) steps
DONRCT
DO 101
IRANIG 106
(b) steps
Figure 10: Comparison of 10,000 samples generated by DDPM, DDIM, and HSIVI-SM.
Table 4: Frobenius distances between the estimated covariance matrices and that of the ground truth. For each method, we collect 100,000 samples to estimate the covariance matrix.
T=
T 3
HSIVI-SM
0.0886
0.0813
0.0431
0.0333
HSIVI-LB 0.0883
0.0825
0.0722
0.0433
ie.
Pbase = =N(x;y,0'1), x= T-1 for 0 T
From Figure 9 that SIVI-LB also underestinates the posterior variance and 5-layer HSIVI-LB fits the variance better. This phenomenon is also observed the performances of SIVI-SM and HSIVI-SM in Figure The quantitative comparison between different numbers of layers is reported in Table where we see that for both HSIVI-SM and HSIVI-LB, the variational approximation gets more accurate with more layers. We also find that HSIVI-SM fits better than HSIVI-LB consistently.
D.3 Toy examples of diffusion model acceleration
We compare the samples from DDPM, DDIM, and our proposed HSIVI-SM with and 10 steps in Figure We find that DDIM and DDPM fail converge to the target distribution with small number of steps, while HSIVI-SM can provide noticeably better samples. Moreover, DDPM tends to underestimate the variance as evidenced by the narrower region occupied by the samples.
D.4 MNIST
Figure 11 shows the samples from DDPM, DDIM, and HSIVI-SM with T = 10 steps. We see that the samples produced by HSIVI-SM is much cleaner and more recognizable than those produced by DDPM and DDIM.
20

---

DDPM (T-10)
DDIM (T-10)
HSIVI-SM (T-10)
3
2
9
8
8
Ω
3
9
2
7
7
8
7
0
3
4
6
7
4
5
9
5 3
57
7
6 6 ?
033
3
4
9
9
6
5
/
5
7
7
7
2
0 4
1
7
4
8
|
22
6
661
2 9
3609
6
327
6
8
6
8
6
22
p
2 2
8
3
3
3
6
9 6
4 6
6 6
2
4
8 0 8
8
o 2 4
/ S
90
9
5
7
9
8
7
4
5
2 5
G
4
1
2
2
6
3
6
4
8
6
2
3
3
Figure 11: Comparison of the quality of uncurated samples generated by DDPM, DDIM, and HSIVI-
SM with 10 discrete time steps on MNIST.
Table 5: Number of parameters in the score model (or noise model) used by different methods in Table 2
'M' refers million.
CIFAR-10
CelebA
other methods
38.72M
78.66M
HSIVI-SM (ours)
38.72M
78.66M
D.5 CIFAR-10 CelebA
Figure 12 shows the uncurated samples from proposed HSIVI-SM method with different numbers of layers on CIFAR- 10 (28 x 28), CelebA (64 x 64) and ImageNet (64 x 64). We also compare the sampling time of different methods when NFE in Figure 13
One can observe that HSIVI-SM has almost same running time as the simplest DDIM algorithm. Finally, we report the number of parameters in the score model (or noise model) used by different methods T which corresponds to Taaleeannd Figure T3 In our implementations of HSIVI- SM, the number of parameters in the noise model equals that in the conditional layer qe gc(x:/<:41;0)) We find that our model with the same parameters reaches better results in Table 2
E
Experimental details
E.1 Target distribution approximation
In this part, set the conditional layer be =x) -Nx [12;; diag(cxp((d)) and the mixing layer to be (O,1I) I) for SIVI. Here, {d" }²) = o are the variational parameters. For T-layer hierarchical semi-implicit variational distribution with T 2, the variational prior qr (wT) set to be (O, I). Each conditional layer gr (21/22441 ④x) for / = o,..., - 1 is a conditional Gaussian distribution
ge (m, [2441; (b)) (ax;;; (Maa:/((aa ( of"), diag (exp((ex)()))))
Note that the o² and TT above are all vectors with the same dimension as æ. We use sequential training for HSIVI in the two experiments in this part. The parameters (ox)) T-11 are independent across different f. If not otherwise specified, we use the Adam optimizer (Kingma Ba 2015) with B (0.9, 0.99) for training.
E.1.1 Gaussian mixture model
For the experiment on the Gaussian mixture model, we construct 5-layer hierarchical semi-implicit variational distributions. The mean of each conditional layer μ1z; g²) in SIVI or p( o in HSIVI has residual form, i.e. 4l(2 σ") + fı(z; g*) and μe( (f dft) = I++1 + f1 oft),
for / = ... T pı g(t) in SIVT and {ī( 1; of )
HSIVI all have the same structures
of multi-layer perceptrons (MLPs) with layer widths [2. 50, 50, 2] and ReI.U activation functions. For each 1,, ff (a:;; v/x) HSIVI-SM and (a; v)) in SIVI-SM are parameterized by MI.Ps with layer widths [2, 128, 128, 2] and ReLU activation functions.
21

---

T=5
1=
T=
(a) CIFAR-IO (32x32)
==5
(b) CelchA (64x64)
T
T
15
ImageNet 64)
Figure 12: Uncurated samples generated by HSIVI-SM with different numbers of layers CIFAR- 10, CelebA and ImageNet.
The noise levels in the diffusion bridge are == 1- for 10, learning rate of variational parameters Φi (or Φ) to 0,001 and the learning rate of th (or 1/) to 0.002 in both SIVI and HSIVI. For HSIVI-LB and HSIVI-SM. we run 80000 variational parameter updates for every conditional layer; for SIVI-LB and SIVI-SM, we run 5X80000 variational parameter updates. For HSIVI-SM and SIVI-SM, in each nested training loop (m vh.) (or f(; v)), we update dh (or vb) one time after each update of di (or Φ). All the algorithms are trained with batch size of 64.
=
4). We set the
F.1.2 High-dimensional conditioned diffusion
For the experiment on high-dimensional conditioned diffusion, we examine the performances of SIVI and 5-layer HSIVI. The ground truth is formed by running 100,000 independent stochastic gradient Langevin dynamics (SGLD) chains with step size of 0.0001 and collecting the results
22

---

CIFAR-10
CelebA
Analybic DPM
ParDDPM
DRM Salver Fast
Analytic-DMM
HSVV-SM louns)
DDIM
DDM
HSM SM tours)
0.50
0.25
0.50
0.75
1.00
1.25
1.50
1.75
Time (seconds)
Time (socands)
Figure 13: Sampling time (1) different methods when NFE 5 on CIFAR- and CelebA. Results averaged by 100 independent runs with batch size of 128 on single Nvidia 2080Ti GPU.
the mean of each conditional layer p( of in HSIVI has a residual form, i.c. 10(x+11; of = 2e+1 + ( (()) For SIVI and t = T - in HSIVI, assume (2(;; 6d) ) = paz;; 6²)) and (x o() 2( ; o( For each t, (3; of') in HSIVI and (2; )²)) in SIVI are MLPs with layer widths [300, 512, 512, 300] and ReLU activation functions. For each fo 4hr) in HSIVI-SM and f(æ: s) in SIVI-SM are MLPs with layer widths [300, 512, 512, 300] and ReLU activation functions. For both SIVI and HSIVI, we train each conditional layer for 100,000 iterations with a batch size of 128. For HSIVI-SM and SIVI-SM, in each nested training loop of f: (mx;; whe) (or f(:; v)), we update the (or v/) one time after each update of Φe (or Φ). We set the learning rate be 0.0001 for de (or Φ) and 0.0005 for th2 (or 15).
after 10,000 iterations. For O....,
E.2 Diffusion model acceleration
In this part, we use the diffusion bridge to construct the auxiliary distributions and joint training as mentioned in Section 4.2 With pre-trained score model noise model, consider the generative tasks as score-based variational inference problems. Therefore, we do not use any training data to train HSIVI-SM.
For HSIVI-SM, the variational prior qr (xT) is set to be (0, I). To avoid the large memory consump- tion, we use the joint training method where the parameters of the conditional layers qe (x, (2,,/<+11 Φ) and (ax;; b) are the same across different t. The t-th conditional layer a conditional Gaussian distribution
qe (=(2||2241: o) = N (ze pa (1141) o*), diag (o cxp()"))
where (6", = á are the variational parameters, d" is vector with the same dimension as æ, and or is a fixed scalar value. We use the generalized inference process in DDIM (Song al. (020a) with the noise level v > to initialize μe (Zt+1; dd) and determine the value of a, for each l. not otherwise specified, we use the Adam optimizer (Kingma & Ba 2015) with 3 = (0.9. 0.99) for training.
For our implementation, we referenced the training code of diffusion model accelcration for our models in the repository from (Dockhorn al. 2022).
E.2.1 Toy examples of diffusion model acceleration
For pre-training the score model S* s), we consider quadratic noise levels 1 1 a(s) = s2 for
E [o, 1]. We then train S* (x, s) on 1000 fixed noise levels (1 (1-a) a(i/1000)) ) 10000 by optimizing the DDPM loss in equation (33) for 200,000 iterations with learning rate of 0.0003 and a batch
12'-1 size of 100. For constructing the diffusion bridge, we choose T discrete time steps { so that
1 - (sx) = [0.01 + (v0.8 - 0.1) for = 0, -I
Model architecture
The model architecture of S* (ax s) is
S [ 8) = MI. (MT.Pembx(ar) (a:) Me.pembt(1 (1 a(s))),
where MIP is a decoder implemented as MLPs with layer widths [128, 128, 128, 2). MLPembx is data embedding block implemented as MLPs with layer widths [2, 128). and MLPembt is a time
23

---

embedding block implemented as MLPs with layer widths [256, 128, 128]. We use the sinusoidal positional embedding (Vaswani et al. 2017) of 1 ass) as the input of MLLpembr All these three MLPs use GELU as activation functions. We use the generalized inference process with noise level
? 1.0 to initialize the conditional layers. The architecture of fi (atri is the same as that of S (æ, s). We initialize fr( v) with S7 (ax) := (z)())
Training setting The learning rate is set to be 0.0002 for qi (2g Φ) and 0.0005 for fi (zx; 1) on Swissroll, Circles, and Moons for both T = 5, 10. On Checkerboard, the learning rate is set to be 0.00001 (0.00002) for train HSIVI-SM for 25,000 iterations with batch size of 64 in all cases. In each nested training loop of ( v), we update v/ 3 times after each update of Φb
()) and 0.00005 (0.0001) for (a vt) when T = 5 (T = 10). We
E.2.2 MNIST
For the experiment on MNIST, we use the pre-trained noise model c" (zz, s) and train HSIVI-SM with c-training introduced in Section C.4 The following construction of noise schedule comes from Song
et al. 2020ap. Let 33 = Amin + Panas Smin j for j = O,..., 999, where miin = 10001,,mma. = 0.02.
999
We pre-train the noise model on the 1000 fixed noise levels - a(s) == I[; 83 for s = 0...., 999
by equation (14). The noise model is trained for 100,000 iterations with learning rate of 0.0001 and
batch size of 64. We then choose T discrete time steps 8 [800 f for t = 0, .. T 1 to construct the T-layer diffusion bridge.
Model architecture The pre- pre-trained noise model e" (zx s) follows the UNet structure employed by Ho et al. (2020) where the number of input channels and output channels is reduced to one. Additionally, pad the image size to 32 x 32 to fit E" exx s). We use the generalized inference process with noise level v = 0.2 to initialize the conditional layers. The architecture of fa v) is the same as that of E ((e,)) s). We initialize (( 16) with -E" (21,,, st)/ / a(s:).
Training setting For both T = 5, 10, the learning rate is set to be 1.6 x 10 for ø and 6.4 x 10-5 for v. We train HSIVI-SM for 10,000 iterations with a batch size of 64 in all cases. In each nested training loop of fa v) we update yb 20 times after each update of o.
E.2.3 CIFAR-10, CelebA & ImageNet
For experiments on CIFAR-10 and CelebA, we use the pre-trained noise model e" (2(x,) s) and train HSIVI-SM with e-training introduced in Section C.4 We use the same noise sched- ule as in the experiment on MNIST. Let 3, = Amin + fmax--Biin for j = 0..... 999, 999
where Smin = 0.0001, 3max and ImageNet seperately scperatcey from ht httpt:tttt//tiiiee /ttt:////ttubb www/tqch////cccccccccccccccccc.t httpt/ghhhhppmmepp/eeeeeepppaatt checkpoints//oe cittitisssssaafaoooeaaar 2040 pt and httpss://ppenaappbbe
0.02. We take the pretrained noise model for CIFARI0 wwwrch/eeeeeeee/ooooooo.eeeeeeee
blob. core. windows. net/ uncond 100M 1500K. pt On CelebA, We pre-train the noise model on the 1000 fixed noise levels 1 ass) := I Bj for s 0,, ...... 9999 by optimizing equation 14]. The noise model is trained for 600 epochs. with learning rate of 0.00002 and batch
diffusion/ on/ maa ch- 2021/imagener/
size of 128. We then choose T discrete time steps 8; 8800 += for t = .,..., T 1 to construct the T-lyeer diffusion bridge.
Model architecture On CIFAR-10 and CelebA, the structure of E" (x(x))) s) is exactly the UNe employed in Ho et al. (2020) without modification; on ImageNet, the structure of e" c"(x, s) is exactly the UNet in Nichol & Dhariwal (22TT). We use the generalized inference process with noise level = 0.2 to initialize the conditional layers. The architecture of f. (xx; is the same as that of e* ''x 8). We initialize f( vhb with -6" (21,
a(s.).
Training setting The number of layers, which also the number of function evaluations (NFE), is set to be T = 5, 10. 15 in our test cases. On CIFAR-10, the learning rate is set to be 1.6 x 10-5
for q 1141; Φ) and 8 x 10-5 5 for (a,;; v);; on CelebA, the learning rate is set be 1.2 x 10-6
for ge(: [ Φ) and 6 x 10 for f. v);; on ImageNet, the learning rate is set to be 1 x 10-5
We use the Pytorch implementation of UNet structure in httpps//ggitubb.ooo///////////p/
24

---

for qi (=+11 σ) and 5 x 10 for fz v). We trained HSIVI-SM for 10,000 iterations with batch size of 128. During each nested training loop of fz 1)), we update 20 times after each update of Ø, since we find (a 1) needs more training empirically to provide reliable guidance. For T 10, 15, we use the above training settings; for T we find that further fine-tuning on the well-trained 15-layer HSIVI-SM for 1,000 iterations yields better results, and we utilize this strategy to optimize the 5-layer HSIVI-SM with a 0. smaller learning rate. Experiments need about 1.5 days on CIFAR-10, 10, need about days CelebA and 4 days on ImageNet using 8 Nvidia 2080 Ti GPUs. During the training, we find that HSIVI-SM converges in the first 30% iterations on CIFAR-10 and converges in the first 50% iterations CelebA.
F
Limitations
For the application of accelerating the sampling process of diffusion models, our HSIVI-SM training involves three models: the score model (or noise model), the conditional layers qr (3, (2++1;0), and fe(r;; v)). As a result, HSIVI-SM requires higher memory consumption due to the involvement of multiple models. Additionally, since our HSIVI algorithm approximates the target distribution using the score function, it necessitates a pre-trained score model (or noise model) with high accuracy and additional training steps. Finally, we recognize that the alternative method HSIVI-LB remains unexplored for accelerating the diffusion model, and we defer this aspect to future research.
25


Reviews:
[{"review_id": "B01pEgSVxn", "reviewer": "Reviewer_T15h", "summary": "This paper presents a new method called Hierarchical Semi-Implicit Variational Inference (HSIVI) that enhances the expressiveness of Semi-Implicit Variational Inference (SIVI) on complex target distributions. HSIVI works by applying SIVI to a hierarchy of latent variables, and using the variational distribution of the previous step as the implicit prior for the later step. The authors apply HSIVI for sampling from diffusion models, learned on a variety of synthetic and real-world datasets.", "questions": "(1) Why isn't the performance of HSIVI-LB reported on the other experiments?\n(2) In terms of computational efficiency, it wasn't clear to me that if two compared methods had the same \"T\", then their computational burden were comparable. Are the computational complexities exactly the same for sampling from HSIVI-SM and SIVI, for equal \"T\"?\n(3) In principle, if HSIVI was used to train the model, we could reach a better data likelihood? Why aren't the authors restricting the application of HSIVI to sampling, and not learning the score function as well?", "limitations": "Yes", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novelty: The paper presents a new method called Hierarchical Semi-Implicit Variational Inference (HSIVI) that enhances the expressiveness of Semi-Implicit Variational Inference (SIVI) on complicated target distributions. This is a novel approach that has not been explored before, to the best of my knowledge.\n\n2. Effectiveness: The authors demonstrate the effectiveness of HSIVI on a variety of synthetic and real-world datasets, including accelerating the sampling process of diffusion models. The results show that HSIVI outperforms existing methods in terms of accuracy and efficiency.\n\n3. Clarity: The paper provides a detailed explanation of the HSIVI method, including its mathematical formulation and implementation. The authors also provide clear and concise descriptions of the experiments and results.\n", "weaknesses": "The main weakness of this work is that the method seem fairly complicated to implement in a practical setting. The method is presented in a general way, with an algorithm box. However, later another objective function is presented with some parameter sharing. Then, in the larger-scale experiments, HSIVI-SM is trained by fune-tuning a larger model. Either the method is not extremely stable / usable, or there are too many details but those are necessary for the experiment to be setup properly? (in the latter case, perhaps one experiment should go to the supplements?).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and valuable feedback. We address your specific questions and comments below.\n\nQ1: The main weakness of this work is that the method seem fairly complicated to implement in a practical setting. The method is presented in a general way, with an algorithm box. However, later another objective function is presented with some parameter sharing. Then, in the larger-scale experiments, HSIVI-SM is trained by fine-tuning a larger model. Either the method is not extremely stable / usable, or there are too many details but those are necessary for the experiment to be setup properly? (in the latter case, perhaps one experiment should go to the supplements?).\n  \nA1: Thanks for your question!\n  \nFirst of all, we propose HSIVI as a general VI method. When used for diffusion model acceleration, we employed parameter sharing to reduce memory consumption so that HSIVI-SM will have the same memory usage as the other baseline methods (e.g., DDPM and DDIM). Moreover, parameter sharing also allows joint training of the conditional layers which would facilitate convergence, similar to parameter sharing of score nets in diffusion models.\n\nNote that the target data distributions in generative models are usually complicated. When $T$ is small, the distance between the auxiliary distributions $\\\\{p_i(x)\\\\}_{i=0}^{T-1}$ would be large, making it challenging to train the conditional layers starting from random initialization. As DDIM can produce reasonably good samples when $T$ is small, it serves as a natural initialization strategy for HSIVI. For the same reason, we used the trained $T=15$ layers HSIVI model to initialize $T=5$ layer HSIVI model. These strategies work well in practice, and we see significant improvement of HSIVI over DDIM (Table 2 in the main text). More details of the experimental setup can be found in Appendix E.2.3. Note that these strategies are more like initialization strategies that can take advantage of current fast diffusion model based samplers and is data free, instead of fune-tuning strategies that often require data samples during training.\n \nWe apologize for the confusion and would clarify these in our revision.\n  \nQ2: Why isn't the performance of HSIVI-LB reported on the other experiments?\n  \nA2: The reason is that only the score function is available in diffusion models, not the probability density function. This makes HSIVI-SM a natural choice as it uses the score matching objective instead of the ELBO related objectives.\n\n  \nQ3: In terms of computational efficiency, it wasn't clear to me that if two compared methods had the same \"T\", then their computational burden were comparable. Are the computational complexities exactly the same for sampling from HSIVI-SM and SIVI, for equal \"T\"? \n\nA3: First, we'd like to clarify that SIVI uses a single conditional layer (i.e., T=1) and hence would be much cheaper to sample from than HSIVI-SM which uses multiple layers $T>1$. For methods that we discussed for diffusion models, the computational complexity would be similar if they had the same $T$. That is because we used the same neural network architecture for the conditional layers in HSIVI and the score nets in diffusion models. We have a comparison of the sampling time of different methods in Figure 12 in Appendix D.5. We will clarify this in our revision.\n\n\nQ4: In principle, if HSIVI was used to train the model, we could reach a better data likelihood? Why aren't the authors restricting the application of HSIVI to sampling, and not learning the score function as well?\n\nA4: Sorry for the confusion! Generally speaking, HSIVI is a variational inference method that assumes the target density is accessible (e.g., the density function up to a constant or the score function is available). When used for diffusion model acceleration, HSIVI-SM does not directly target the generative model. Instead, it requires a sequence of auxiliary distributions that bridges between a simple distribution and the target distribution which is available given the learned score functions of diffusion models (Example 2). Note that this also allows data-free training of HSIVI-SM to accelerate diffusion model sampling. Learning the score functions together with HSIVI is an interesting idea, and would allow for a better forward process. However, it would also be more challenging as the score functions required by HSIVI now need to be learned together with the conditional layers of hierarchical semi-implicit distributions."}, {"review_id": "CbEVqdxYZt", "reviewer": "Reviewer_i7TH", "summary": "The paper introduces a hierarchical semi-implicit variational inference that stacks multiple semi-implicit layers to construct a flexible generative distribution. The paper introduces a sequence of distributions that interpolate between the target and a base distribution. Each pair of intermediate inference and target distributions are matched by optimizing a SIVI bound. The method is applied in the diffusion model for sampling acceleration. \n\n##Post-rebuttal\n\nThanks for the authors' response. Most of my questions have been addressed and I will maintain my current score.", "questions": "- To train HSIVI sequentially, the paper mentions, “Let the parameters ϕ_t in the t-th conditional layer be independent across different ts.” But since q_t(x_t; ϕ_t) depends on x_{t-1} that depends on ϕ_{t-1}, It is understandable that the graph detaching in the algorithm benefits computation efficiency. Does it have any downside for not training ϕs jointly? For example, will there be convergence issues, or will some ϕs overfit while others underfit? \n\n- In Appendix C.3., the author shows that the detaching operation keeps the optimality of \\phi unchanged in the joint training. Does a similar conclusion apply to the sequential training in the main paper?\n\n- Table 1 shows with the same layers/steps, HSIVI achieves better approximation than DDPM and DDIM. Would the computation with one layer of HSIVI be more than one step of DDPM?  If so, it might not be a fair comparison. Can the author discuss the computational complexity and provide wall clock time? \n\n- [1] and [2] design a hierarchical model using semi-implicit distributions, which seem to be closely related to HSIVI. The paper might need more detailed discussions of these papers. \n\n[1] Importance weighted hierarchical variational inference (NeurIPS 2019)\n[2] Structured Semi-Implicit Variational Inference (AABI 2019)", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of using semi-implicit distribution for the progressive approximation is interesting and intuitive. \n\n- The empirical results show HSIVI improves the distribution approximation and sample quality significantly on several tasks. \n\n- Though the modeling and training of HSIVI consists of many components, the paper explains the ideas concisely and clearly.", "weaknesses": "- The validity of the training procedure needs more concrete explanations or theoretical results\n\n- The empirical study needs more evidence for the improvement of acceleration.\n\n- Some closely related works need more discussions of similarity and improvement. \n\nPlease see Questions for more details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and helpful feedback! We address your concerns as follows.\n\nQ1: The validity of the training procedure needs more concrete explanations or theoretical results.\n\nA1: The training procedure is similar to SIVI-SM (https://openreview.net/forum?id=sd90a2ytrt), with the only difference being using multiple auxiliary distributions instead of the target distribution directly. \n\nQ2: The empirical study needs more evidence for the improvement of acceleration.\n\nA2: Thank you for your suggestion! We have provided additional results on CelebA (64x64), ImageNet (64x64) with more powerful pre-trained score nets. These results show that our methods work well for common benchmark datasets for diffusion models and can benefit from a more accurate score function. For CelebA (64x64) experiments here, we adopt **the identical Unet in [3] with 78.66M parameters, which is bigger than the one we used in the main text** with one more downsampling block and one more upsampling block. For ImageNet (64x64), we follow DPM-solver, taking the same improved UNet in [4], which is larger than the one in [3] and has 115.47M parameters.\n\nTable: Sample quality measured by FID on CelebA (64x64). All methods without \\* employ the same UNet in [3].\n|Model\\NFE|5|10|15|\n|----|----|----|----| \n|DDPM|366.10|309.95|206.92|\n|DDIM|27.38|10.89|7.78|\n|FastDPM|27.63|15.44|12.05|\n|Analytic-DDPM|50.92|28.93|21.84|\n|Analytic-DDIM|29.40|15.74|12.25|\n|DPM-Solver-fast|355.96|6.76|2.98|\n|**HSIVI-SM (ours), in the main text (smaller UNet)\\***|8.29|4.95|4.66|\n|**HSIVI-SM (ours)**|**6.22**|**3.09**|**2.23**|\n\nTable: Sample quality measured by FID on ImageNet (64x64). All methods employ the same UNet in [4].\n|Model\\NFE|5|10|15|\n|----|----|----|----| \n|DDPM|402.68|358.80|284.00|\n|DDIM|147.03|42.31|24.85|\n|Analytic-DDPM|N/A|60.65|45.98|\n|Analytic-DDIM|N/A|70.62|41.56|\n|DPM-Solver-fast|402.43|28.96|20.03|\n|**HSIVI-SM (ours)**|**40.43**|**17.67**|**15.49**|\n\n[3] Denoising diffusion probabilistic models. NeurIPS 2020.\\\n[4] Improved denoising diffusion probabilistic models. ICML 2021.\n\nQ3: Some closely related works need more discussions of similarity and improvement.\n\nA3: We will add discussions on related works in our revision. Please see more details in A7.\n\nQ4: To train HSIVI sequentially ... Does it have any downside for not training $\\phi$s jointly? For example, will there be convergence issues, or will some $\\phi$s overfit while others underfit?\n\nA4: Thanks for this interesting question! In the current setting, each conditional layer is trained to push the fitted distribution from the previous layers toward the auxiliary distribution at the next time step, and then it is fixed afterward. The reason is that the trained $q_t(x_t; \\phi\\_{\\ge t})$ would provide a good approximation to the corresponding auxiliary distribution $p_t(x_t)$ and hence reduce the difficulty for training the next conditional layers (since the differences between successive auxiliary distributions are assumed to be small). Note that this approach does not require each $q_t(x_t; \\phi_{\\ge t})$ to be a perfect match of $p_t(x_t)$, as the conditional layer at time $t-1$ would automatically compensate for the error from previous time steps when fitting $p_{t-1}(x_{t-1})$ (see an example in Figure 7 in the appendix). Therefore, we do not expect convergence issues given enough capacity of the conditional layers and an appropriate $T$. On the other hand, when $\\phi$s are trained jointly, there is a potential drawback that the previously fitted $q_t(x_t; \\phi\\_{\\ge t})$ would deviate from the corresponding auxiliary distribution, and this would make the training less stable as well (see Figure 1 in the rebuttal PDF). We will add a discussion to our revision.\n\nQ5: In Appendix C.3., the author shows that the detaching operation keeps the optimality ... Does a similar conclusion apply to the sequential training in the main paper?\n\nA5: Yes, this result holds true as well. In sequential training, we assume that the marginal distribution of the variational distribution is fitted to the target distribution $p_{t+1}$. This is used as a mixing layer during the training of $q_t(x_t|x_{t+1};\\phi_t)$, where we apply a detaching operation to keep the parameters $\\phi^\\star_{\\ge t+1}$ fixed. As a result, the optimality of the model is maintained until the final $t=0$. We will include these details in Appendix C.3.  \n\n\nQ6: Would the computation with one layer of HSIVI be more than one step of DDPM? If so, it might not be a fair comparison. Can the author discuss the computational complexity and provide wall clock time?\n\nA6: The computation with one layer of HSIVI is about the same as one step of DDPM/DDIM as we used the same neural network architecture for the conditional layers in HSIVI and the score nets in diffusion models. A wall clock time comparison is provided in Figure 12 in the appendix.\n\nQ7: [1] and [2] design a hierarchical model using semi-implicit distributions, which seem to be closely related to HSIVI. The paper might need more detailed discussions of these papers.\n\nA7: Thank you for sharing these relevant papers. We will add more detailed discussions of these papers in our revised manuscript. (i) IWHVI [1] employs a reverse model as an importance distribution for the variational distribution and requires an explicit variational prior. Our proposed HSIVI inherits the advantage of SIVI that allows $q_t(x_t;\\phi_{\\geq t})$ to be implicit and does not require a reverse model. These properties of HSIVI improve its expressiveness and enable multi-layer extension. (ii) Structured SIVI [2] assumes an autoregressive form of variational distributions to factorize high-dimensional joint semi-implicit distribution into the product of low-dimensional conditional semi-implicit distributions. Our approach is distinct from structured SIVI because HSIVI does not factorize the data space but augments it with auxiliary bridges to accommodate the multimodal targets."}, {"review_id": "qj4qrHdLh7", "reviewer": "Reviewer_PvNK", "summary": "The authors extend semi-implicit variational inference to have multiple layers of latent variables, vastly increasing the. expressiveness of the model. A nice formulation of the asymptotic lower bound plus a training scheme is introduced as well.", "questions": "I don't have any questions.", "limitations": "It seems like the main text is missing a limitations section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is an extension that on the surface seems straight forward. But the authors demonstrate that this is far from the case and introduce a novel formulation of the semi-vi ELBO.\n\nThe paper was also written well and easy to follow. The experiments section was great too. ", "weaknesses": "My only qualm is the flow between the first half of the paper and the second half where the focus is on diffusion models. Previously, the authors argued that training using a sequence of distributions would lead to better marginal distributions. But for speeding up sampling of diffusion models, it seems like training the joint distribution leads to impressive results. This opens up the question of whether the sequence of distributions is even needed at akk.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and valuable feedback! Below are our answers to your comments:\n\nQ1: My only qualm is ... But for speeding up sampling of diffusion models, it seems like training the joint distribution leads to impressive results. This opens up the question of whether the sequence of distributions is even needed at all.\n\nA1: Thank you for the discussion on this issue! We want to clarify a potential confusion here regarding \"joint training\" and \"training the joint distribution\" in the diffusion model acceleration part. In our proposed HSIVI approach, only the marginal distributions are fitted, not the joint distributions (we have a related discussion in Section 3.2). The phrase \"joint training\" is introduced when we use parameter sharing to reduce memory usage which is a common practice in diffusion models (e.g., score nets). Since the conditional layers now are parameterized with the same $\\\\phi$, they would be trained jointly, instead of being independently trained as in the sequential training case where each layer has its own parameters. Therefore, \"joint training\" is used to indicate a training behavior and it is the marginal distribution $p_t(x_t), t=0,\\\\ldots, T$ given by a sequence of auxiliary distributions that are approximated progressively with the hierarchical semi-implicit distributions. We see this marginal approximation property as one of the keys that allow HSIVI to generate high-quality samples with small function evaluations (see the discussion paragraph in section 3.2, i.e. line 150-157, for more details).\n  \n\nQ2: It seems like the main text is missing a limitations section.\n\nA2: Due to the space constraints, we have placed the limitations section in the appendix. We will move the limitations section to the main text in our revision if there is room."}, {"review_id": "0fruVbRRK7", "reviewer": "Reviewer_EivT", "summary": "Authors propose a hierarchical semi-implicit variational inference framework by extending the existing SIVI. Authors showed that the proposed HSIVI, given pre-trained score networks, can be used to accelerate the sampling process of diffusion models with the score matching objective. The numerical results show more enhanced performance in faithful modeling of complex distributions and diffusion model acceleration.", "questions": "- It is mentioned in the manuscript that HSIVI requires more memory. How much more memory (with respect to data dimension) does it need compared to baselines? \n- ImageNet (64x64) is a common benchmark dataset for diffusion models. I would be nice to have the results for that as well.\n- How does the proposed model performs compared to Diffusion Normalizing Flows [1] both in terms of target distribution approximation and sample generation.\n- [A suggestion] It would be interesting to extend HSIVI on graphical models (in a setup like [2]).\n\n\n\n[1] Diffusion Normalizing Flow, Qinsheng Zhang and Yongxin Chen, NeurIPS 202.\n[2] Efficient Inference Amortization in Graphical Models using Structured Continuous Conditional Normalizing Flows, Christian Weilbach, 2019.", "limitations": "The limitations to some extent have been discussed but more discussion (specially around memory usage) is needed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is fairly well-written, clearly motivated and generally addresses an important problem. The extension of SIVI to a hierarchical model, although not very novel, is intuitive and makes sense. I haven't checked the proofs but the overall approach seems to hold up.", "weaknesses": "- Extending SIVI to a hierarchical model, by itself, is not very novel. However, the application to diffusion acceleration is interesting. \n- The experiments and presented numerical results could be improved. (check next section for more)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful feedbacks and suggestions! Here are our responses to them.\n\nQ1: Extending SIVI to a hierarchical model, by itself, is not very novel. However, the application to diffusion acceleration is interesting.\n\nA1: Thank you for acknowledging the novelty of the application to diffusion acceleration! We admit that HSIVI is a natural extension of the SIVI framework. However, we also believe that HSIVI's contributions go beyond SIVI in the following aspects:\n\n1. HSIVI constructs more expressive mixing layers using multi-layer architectures. Standard SIVI only has one conditional layer. We show that this may not be enough for distributions with complicated structures (see Figure 2 and 3 in Section 5.1). By allowing multiple conditional layers, HSIVI further improves the flexibility of semi-implicit distributions.\n\n2. HSIVI introduces an auxiliary bridge to alleviate the difficulty of fitting the target distribution. Training HSIVI, therefore, can be done in a sequential manner, where the intermediate semi-implicit distributions were pushed towards the target distribution layer after layer. Note that this is only possible in the context of semi-implicit variational inference, where the mixing layer is allowed to be implicit, and this sets HSIVI apart from the previous methods like hierarchical variational models [4]. These auxiliary distributions would also anchor the intermediate semi-implicit distributions $q\\_t(x\\_t), t=T-1, \\ldots, 1$, making the training process more stable.\n\nQ2: The experiments and presented numerical results could be improved. (check next section for more)\n\nA2: We have new results on ImageNet (64x64) that compare favorably to other baselines (see A4 below). We will add these new results and other relevant baseline methods (e.g., Diffusion Normalizing Flows [1]) to the experiments in our revision.\n\nQ3: It is mentioned in the manuscript that HSIVI requires more memory. How much more memory (with respect to data dimension) does it need compared to baselines?\n\nA3: In addition to the score nets as in standard diffusion models, HSIVI requires the conditional layers $q_t(x_t|x_{t+1};\\\\phi), t=T-1,\\\\ldots, 0$ and $f_t(x_t;\\\\psi), t=T-1,\\\\ldots, 0$ for training, when parameter sharing is applied. These two additional parts both take the same memory consumption as the score nets. Therefore, the memory consumption of HSIVI during training is about three times of those baseline methods. After training, both the score network and $f_t(x_t;\\\\psi), t=T-1,\\\\ldots, 0$ can be removed, and only the conditional layers $q_t(x_t|x_{t+1};\\\\phi), t=T-1,\\\\ldots, 0$ are needed for sampling from HSIVI. So during sampling, the memory consumption of HSIVI is about the same as the baseline methods.\n \nQ4: ImageNet (64x64) is a common benchmark dataset for diffusion models. I would be nice to have the results for that as well.\n\nA4: We conducted additional experiments on the ImageNet (64x64) dataset, which demonstrated that HSIVI also achieves significant acceleration to diffusion model sampling at 5, 10, and 15 steps.\n\nTable: Sample quality measured by FID(&#8595;) on ImageNet (64x64). All methods employ the same UNet in [3].\n| Model\\NFE | 5  | 10 | 15 |\n| ---- | ---- | ---- | ---- | \n| DDPM  | 402.68 | 358.80 | 284.00 |\n| DDIM  | 147.03 | 42.31 | 24.85 |\n| Analytic-DDPM  | N/A | 60.65 | 45.98 |\n| Analytic-DDIM | N/A |  70.62 | 41.56 |\n| DPM-Solver-fast | 402.43 | 28.96 | 20.03 |\n| **HSIVI-SM (ours)** | **40.43** | **17.67** | **15.49** |\n\n\nQ5: How does the proposed model performs compared to Diffusion Normalizing Flows [1] both in terms of target distribution approximation and sample generation.\n\nA5: Thank you for mentioning this relevant work. Unlike standard diffusion models, Diffusion Normalizing Flows (DiffFlow) allow learnable forward SDE as well and are trained by matching the forward and backward SDEs [1]. Compared to standard diffusion models, DiffFlow would require fewer discretization steps and thus has better sampling efficiency. In contrast, HSIVI is more like standard diffusion models that use a fixed forward process, while achieving better sampling efficiency via variational approaches using hierarchical semi-implicit distributions. Therefore, the training of HSIVI only requires a pre-trained score function and can be data-free.\n \nDiffFlow can be used for density estimation and sample generation. As HSIVI is not designed for density estimation, we compare HSIVI to DiffFlow in terms of sample generation on CIFAR-10 as follows: \n\nTable: Sampled quality measured by FID(&#8595;) on CIFAR10 (32x32). \n| NFE  | DDPM| DDIM| DiffFlow | HSIVI-SM   |\n|--------|------|------|------|------------|\n| 5      | 320.16   | 41.53   | 28.31   | **6.27**  |\n| 10     | 278.65   |13.73   | 22.56   | **4.31**  |\n\nWe will add these results in our revision.\n\n[1] Zhang, Qinsheng, and Yongxin Chen. \"Diffusion normalizing flow.\" in NeurIPS 2021.\n\nQ6: [A suggestion] It would be interesting to extend HSIVI on graphical models (in a setup like [2]).\n\nA6: Thank you for the suggestion! Extending HSIVI to graphical models is indeed very interesting. We will read it carefully and think about possible extensions.\n \nQ7: The limitations to some extent have been discussed but more discussion (specially around memory usage) is needed.\n\nA7: We will add more discussion on memory usage in our revision, please see A3 for more details.\n\n[1] Zhang, Qinsheng, and Yongxin Chen. \"Diffusion normalizing flow.\" in NeurIPS 2021.\\\n[2] Weilbach, Christian, et al. \"Efficient inference amortization in graphical models using structured continuous conditional normalizing flows.\" in AABI 2019.\\\n[3] Nichol, Alexander Quinn, and Prafulla Dhariwal. \"Improved denoising diffusion probabilistic models.\" in ICML 2021.\\\n[4] Ranganath, Rajesh, Dustin Tran, and David Blei. \"Hierarchical variational models.\" in ICML 2016."}, {"review_id": "vJKM4UybUz", "reviewer": "Reviewer_MCSt", "summary": "Semi-implicit variational inference increases the expressiveness of variational posteriors but introduces intractabilities to their inference. This work adds to the semi-implicit variational inference (SIVI) work of Yu and Zhang (https://openreview.net/forum?id=sd90a2ytrt) which uses an alternative objective (the Fisher divergence) that, combined with the hierarchical nature of semi-implicit variational distributions, can be approximated as the solution of a mini-max optimization problem. This work takes inspiration from simulated annealing and denoising-diffusion models by proposing to expand the single-layer semi-implicit variational family into multiple-layer variants by specifying hierarchical auxiliary distributions to guide the semi-implicit variational distribution towards the target distribution. The authors denote this method Hierarchical Semi-Implicit Variational Inference (HSIVI). The auxiliary distributions are specified either by assuming the marginal densities of a geometric interpolation between a target and base distribution are available analytically or by using pre-trained scores from a diffusion bridge. The main contribution to this paper is stated in the author's claim that \"when used for diffusion model acceleration, we show that HSIVI can produce high quality samples comparable to or better than the existing fast diffusion model based samplers with small number of function evaluations on various datasets.\" This claim is tested empirically on the CIFAR-10 (32x32) and CelebA (64x64) datasets with results showing comparable or better FID scores to existing fast diffusion model samplers with few (between 5 and 15) function evaluations. As well as the main contribution, an algorithm for training is provided by noting that the conditional layers can be trained sequentially until convergence by exploiting the hierarchical structure of the semi-implicit distribution. Also, an efficient parameterization of the neural networks is provided to make the algorithm computationally feasible.", "questions": "1. It would be interesting if the authors could demonstrate a failure case for their hierarchical variational posterior approximating the denoising-diffusion model, what properties of the score in the backwards process (equation 9) would cause the proposed variational posterior to fail with a small number of function evaluations?\n\n2. No interpretation of the neural network f_t(x_t) is given, apart from that it provides \"guidance\". Could the authors please clarify what is being guided, and to what?\n\n3. Comparing the proposed sampling algorithm to denoising-diffusion: the sampling algorithm for denoising-diffusion uses a reverse-time Markov chain that has specified isotropic variance that appears due to the assumed forward-time transition densities. Whereas the proposed HIVI sampling algorithm in this work is a Markov chain with transition densities that are the conditional variational posterior distributions (called the conditional layer in this work).  The difference between this Markov chain and the denoising-diffusion one is the trainable non-isotropic variance of the conditional layer q_t(\\cdot|x_{t+1}; \\phi). If the conditional layer is instead modelled as a Gaussian distribution with mean and an isotropic scale-Identity matrix, then do you expect that there is no improvement over denoising-diffusion with the same number of steps? It would be good to see an experiment that showed the trade off between sample quality and number of function evaluations using this simpler variational family, which is on par with the denoising-diffusion sampling.\n\n\nTypos:\n\nFisher divergence is not symmetric in its arguments and has been incorrectly written with arguments transposed (the expectation is taken over the distribution of the first argument).\nSee for example how the Fisher divergence is written in the papers that this work has cited\nhttps://arxiv.org/pdf/1602.03253.pdf\nor\nhttps://arxiv.org/pdf/1810.03545.pdf\n\nLine 510, 511, 512 there are typos in the norm regularizing the f network (written as f_{t}(x; \\psi_t) and should be f_{t}(x_{t}; \\psi_t))", "limitations": "The limitation discussion picks up on the important points, but it would be good to know if higher dimensional problems were attempted, or problems with more parameters (such as the \"huge VP deep continuous-time model (Song et al., 2020b) that has more channels and layers\"), and did the authors run into memory issues that were discussed in Appendix F?\n\nAn ablation of the variational family for the conditional layers (as described in question 3) comparing directly to the denoising-diffusion sampling remains unexplored, and would be useful to see.\n", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "Quality\n\nThe work is a novel combination of well-known, peer-reviewed techniques including simulated annealing and SIVI. It is clear how this work differs from the work of Yu and Zhang (https://openreview.net/forum?id=sd90a2ytrt) using the hierarchical semi-implicit variational posteriors, and the value of this contribution is demonstrated on a multi-modal problem in Figure 2. The methods used seem to work well judging by the experimental results. \n\nClarity\n\nThe submission is well-organized and clearly written.", "weaknesses": "Originality\n\nThere is limited technical novelty in this paper as the technical content on the work is almost entirely covered by the work of Yu and Zhang (https://openreview.net/forum?id=sd90a2ytrt) that rewrites the Fisher divergence leading to a form that does not require computing the score of the hiearchical variational posterior. The paper extends this by sequentially score matching to noised versions of the target distribution, which have been obtained analytically or approximated from pretrained denoising diffusion models. The main novelty in this work is instead an experimental one, showing that HSIVI can produce high quality samples comparable to or better than the existing fast diffusion model based samplers with a small number of function evaluations on two datasets by comparing FID scores (CIFAR-10 (32x32) and CelebA (64x64)) and one dataset by comparing samples visually (MNIST). The experiments were made computationally feasible by a trick to parameterize the neural networks that is detailed in Proposition 1.\n\nQuality of results\n\nRelatively simple experiments are provided that on one hand demonstrate the contribution well, but on the other hand do not cover a very diverse range of cases. No empirical failure case is provided for a complicated distribution that requires many function evaluations to sample from, which may provide an interesting avenue for future work.\n\nOn the reproducibility of the results, whilst the experimental setups are well detailed, no code is provided with the paper, nor indication it will become available upon publication. The reviewer would like to see that the code is made available since this is crucial for reproducibility. Furthermore, there may not be enough information to replicate the experiments. For example, as far as I can see, there is no discussion on the parameterization/values of the positive weighting function, beta(t) for joint HSIVI-SM training on CIFAR-10 and MNIST, which makes it impossible to reproduce without any code provided.\n\n\nSignificance\n\nThe work's significance would be improved if it addressed why we should expect improvements in sampling efficiency over denoising-diffusion sampling and where these improvements can be derived from (the particular form of variational family, or something else?).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and valuable feedback. We will address your concerns in the following aspects:\n### Weaknesses\n#### Originality\nAnswer: First, HSIVI not only extends SIVI by allowing multiple conditional layers which greatly enhances the flexibility but also introduces a sequence of auxiliary distributions for progressive training that further improves stability. Note that this is only possible in the context of semi-implicit variational inference, where the mixing layer is allowed to be implicit, and this sets HSIVI apart from the previous methods like hierarchical variational models [1]. Moreover, using HSIVI for diffusion models is more than just an experimental one, and there are also significant methodological contributions. Note that most of the current approaches for accelerating diffusion models are ODE-based, which could limit the diversity of generated samples. The HSIVI approach is different from these acceleration techniques by directly accelerating the SDE. We show in Figure 6 that HSIVI can generate diverse samples even if the starting state is the same. \n\n[1] Hierarchical variational models. in ICML 2016.\n\n#### Quality of results\nAnswer: Thank you for your suggestions. We provide a failure case in Figure 2 of the rebuttal PDF, demonstrating that the HSIVI-SM algorithm fails when the layer number $T$ is small (the distances of successive auxiliary distributions are large) on a checkerboard distribution. In fact, the score function on the checkerboard target is sharp on the boundaries but vanishes elsewhere. Therefore, fitting this target distribution is somewhat challenging. \n\nRegarding reproducibility, the parameterization of beta(t) and other details can be found in Appendix E.2.2. We also have sent an anonymous link of codes to AC.\n\n#### Significance\nAnswer: The improvement in sampling efficiency over denoising-diffusion sampling comes from the expressiveness of semi-implicit variational families. More specifically, we have shown (at least experimentally) that using the same neural network architecture as the score nets, the semi-implicit variational distribution can be used to compress many steps of SDE simulations in DDPM by fitting toward the score functions at the corresponding times (i.e. the score functions in the diffusion bridge in Example 2). \n\nAnother point of view is from the training objective. Note that DDPM matches the joint distributions by minimizing $KL(p(x_{0:T−1})|q(x_{0:T−1}))$. When $T$ is small, the variational distribution $q(x_{0:T−1})$ may be insufficient to approximate $p(x_{0:T−1})$ well enough, leading to degraded marginal distribution approximations of $q_0(x_0)$ to $p_0(x_0)$. In contrast, HSIVI matches the marginal distributions $q_t(x_t)$ and $p_t(x_t)$ and hence would ensure a better fit for $p\\_0(x\\_0)$ especially when $T$ is small.\n### Questions\nQ1: It would be interesting if the authors could demonstrate a failure case ...\n\nA1: Please refer to the answer of Weakness-Quality above.\n\nQ2: No interpretation of the neural network f_t(x_t) is given ... \n\nA2: The role of $f_t(x_t)$ is to approximate $\\nabla \\log p_t(x_t) - \\nabla \\log q_t(x_t)$, thereby identifying regions where the current variational distribution $q_t(x_t)$ fits the target marginal distribution $p_t(x_t)$ insufficiently. We visualized the training dynamics of $f_0(x_0)$ on the checkerboard target in Figure 3 of the global rebuttal PDF. \n\nQ3:  It would be good to see ... using this simpler variational family, which is on par with the denoising-diffusion sampling.\n\nA3: We train HSIVI-SM with isotropic conditional layers on par with DDPM on CIFAR-10 (see the following table).\n\nTable: Sampled quality measured by FID on CIFAR10 (32x32). \n|NFE\\Model|DDPM|DDIM|HSIVI-SM (isotropic)|HSIVI-SM (non-isotropic)|\n|----|----|----|----|----|\n|5|320.16|41.53|7.33|**6.27**|\n|10|278.65|13.73|4.78|**4.31**|\n|15|198.00|8.78|4.46|**4.17**|\n\nThe above results also support the findings discussed in Significance. The improvement of HSIVI-SM over DDPM stems not only from a more expressive variational distribution but also from the direct matching of the marginal distributions.\n\nQ4: Typos\n\nA4: Thanks for catching the typos. We will correct these typos in our revision.\n\n### Limitations\nQ5: It would be good to know if higher dimensional ... and did the authors run into memory issues that were discussed in Appendix F?\n\nA5: We have implemented HSIVI-SM using the score function in [1] on ImageNet64, which demonstrates that HSIVI can also handle high dimensional problems with large models. \n\nTable: Sampled quality measured by FID on ImageNet (64x64). All these methods employ the same UNet in [1] with 115.47M parameters.\n|Model\\NFE|5|10|15|\n|----|----|----|----|\n|DDPM|402.68|358.80|284.00|\n|Analytic-DDPM|-|60.65|45.98|\n|Analytic-DDIM|-|70.62|41.56|\n|DDIM|147.03|42.31|24.85|\n|DPM-solver-fast|402.43|28.96|20.03|\n|**HSIVI-SM (ours)**|**40.43**|**17.67**|**15.49**|\n\nIn fact, we also find that HSIVI can benefit from more accurate score networks (larger models).\nThe score networks on CelebA (64x64) that we used in the main text is a relatively small (see the footnote in page 8). We have used a larger UNet in [2] and got better results (see the table below).\n\nTable: Sampled quality measured by FID on CelebA (64x64). \n|Model|5|10|15|\n|----|----|----|----|\n|HSIVI-SM(UNet with 38.72M parameters)|8.29|4.95|4.6|\n|HSIVI-SM(UNet with 78.66M parameters)|**6.22**|**3.09**|**2.23**|\n\nIn fact, the memory usage in HSIVI-SM during training is about three times of those baseline methods. We did not run into memory issues as the memory usage is still in an acceptable range. More details are presented in our released code.\n\n[1] Improved denoising diffusion probabilistic models. in ICML 2021.\\\n[2] Denoising diffusion probabilistic models. in NeurIPS 2020.\n\nQ6: An ablation of the variational family for the conditional layers ... to see.\n\nA6: We have implemented HSIVI with isotropic conditional layers (in A3)."}, {"review_id": "FouEuhDExP", "reviewer": "Reviewer_AGbR", "summary": "The paper introduces hierarchical semi-implicit variational inference (HISIVI) as an extension of semi-implicit variational inference (SIVI). HISIVI incorporates interpolating distributions between prior and target data to train conditional layers progressively, resulting in accelerated sampling in diffusion models.", "questions": "How does the proposed optimization for score function in HISIVI outperform or differ from the conventional implicit score matching or denoising score matching[7]?\n\n[7] Vincent, P., 2011. A connection between score matching and denoising autoencoders. Neural computation, 23(7), pp.1661-1674.", "limitations": "This work only deals with the geometric interpolation of distributions without providing justification. Exploring the impact of choice of interpolation on generation would enhance the quality of this work.", "rating": 3, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 1, "strengths": "The paper is well-structured and easy to follow.", "weaknesses": "- The novelty of HISIVI compared to related work is not clearly demonstrated. It is recommended to include more related works and comparison: cascaded diffusion models[1], diffusion schrodinger bridge[2], flow matching[3], or rectified flow[4], etc.\n- The experiment section is relatively weak. Comparisons could strengthen the evaluation (aforementioned references and more [5],[6], etc.)\n\n[1] Ho, J., Saharia, C., Chan, W., Fleet, D.J., Norouzi, M. and Salimans, T., 2022. Cascaded diffusion models for high fidelity image generation. The Journal of Machine Learning Research, 23(1), pp.2249-2281.\n[2] De Bortoli, V., Thornton, J., Heng, J. and Doucet, A., 2021. Diffusion Schrödinger bridge with applications to score-based generative modeling. Advances in Neural Information Processing Systems, 34, pp.17695-17709.\n[3] Lipman, Y., Chen, R.T., Ben-Hamu, H., Nickel, M. and Le, M., 2022. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747.\n[4] Liu, X., Gong, C. and Liu, Q., 2022. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003.\n[5] Xiao, Z., Kreis, K. and Vahdat, A., 2021. Tackling the generative learning trilemma with denoising diffusion gans. arXiv preprint arXiv:2112.07804.\n[6] Zheng, H., Nie, W., Vahdat, A., Azizzadenesheli, K. and Anandkumar, A., 2022. Fast sampling of diffusion models via operator learning. arXiv preprint arXiv:2211.13449.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback! Here are our response to your concerns.\n\nQ1:  The novelty of HSIVI compared to related work is not clearly demonstrated. It is recommended to include more related works and comparison: cascaded diffusion models [1], diffusion schrodinger bridge [2], flow matching [3], or rectified flow [4], etc.\n\nA1: First, we'd like to clarify the novelty of HSIVI compared to related works on diffusion model acceleration. (i) HSIVI can be trained without data. As a variational approach, HSIVI uses flexible hierarchical semi-implicit distributions and introduces a sequence of auxiliary distributions for progressive training. For diffusion models, the pre-trained score networks serve as a natural sequence of auxiliary distributions (i.e., diffusion bridge, see Example 2) for HSIVI. \n(ii) Note that most of the current approaches for accelerating diffusion models are ODE-based, which could limit the diversity of generated samples. The HSIVI approach is different from these acceleration techniques by directly accelerating the SDE. We show in Figure 6 that HSIVI can generate diverse samples even from the same starting state.\n\nThanks for your recommendation! See below for a comparison of HSIVI with these related works.\n\n  - Cascaded diffusion model [1] comprises a cascaded pipeline of diffusion models which generate high-resolution samples by successive upsampling from low-resolution samples. Our HSIVI is clearly distinct from [1] in that we aim at accelerating diffusion sampling by assuming a diffusion bridge between Gaussian prior and data distribution.\n  - Diffusion Schrödinger bridge (DSB) [2] reformulates the generative modeling as a Schrödinger bridge problem in finite time and proposes an approximate IPF algorithm that requires recursively solving the half-bridge problem that matches the joint path distributions, as opposed to HSIVI that fits the marginal distributions.\n  - Flow matching [3] and rectified flow [4] learn ODE based generative models by regressing the vector fields of fixed conditional probability paths (which could be made straight by connecting the samples from the prior and target distributions and constructing a deterministic coupling). HSIVI differs from them in that HSIVI uses pre-trained score networks (and therefore is data-free) and directly accelerates the SDE.\n  - Denoising Diffusion GANs [5] proposes using GANs to model each sampling step and optimizing them through an adversarial loss with data. In constant, we assume an explicit form (e.g., Gaussian) of the conditional layers which is optimized via a VI approach without data.\n  - [6] proposes a neural operator, which learns the trajectories of the probability flow ODE, and allows for high-accuracy simulation in a few steps. This idea can be thought of as a data-driven distillation for ODE. In contrast, HSIVI originates in a variational inference perspective which is trained without data and directly accelerates the SDE.\n\nQ2: The experiment section is relatively weak. Comparisons could strengthen the evaluation (aforementioned references and more [5],[6], etc.)\n\nA2: We compare the HSIVI-SM with more baselines on CIFAR10 (32x32). We want to clarify that as a data-free method, HSIVI-SM inherently faces the difficulty of lacking true samples. Also, due to the extra stochasticity, compressing SDE is generally more challenging than compressing ODE. \n\nTable: Sample quality measured by FID on CIFAR10 (32x32). \n|Model|5| 10| 15|\n|----|----|----|----|\n|DDPM|320.16|278.65|198.0|\n|FastDPM|67.64|9.85|6.1|\n|Analytic-DDPM|93.16|34.54|20.0|\n|Analytic-DDIM|51.86|14.08|8.6|\n|DDIM|41.53|13.73|8.7|\n|DPM-solver-fast|329.13|10.89|4.67|\n|DiffFlow [7] |28.31|22.46 |N/A    |\n|HSIVI-SM|**6.27**|**4.31**|**4.17**|\n\nTable: Sample quality measured by FID on CIFAR10 (32x32) for the other results with different architectures. \n|Model|FID|\n|----|----|\n|FM[3]| 6.35 (142 NFE)|\n|DDGAN[5]|3.75 (4 NFE) |\n|2-Rectified Flow[4] |3.36 (110 NFE)|\n|2-Rectified Flow[4] |12.21 (1 NFE)|\n|2-Rectified Flow(+Distill)[4]|4.85 (1 NFE)|\n\n[7] Diffusion normalizing flow. NeurIPS 2021.\n\nQ3: How does the proposed optimization for score function ... denoising score matching?\n\nA3: As a variational approach, HSIVI, or more specifically, HSIVI-SM that uses a score matching objective (i.e., Fisher divergence), assumes a target score function, instead of estimating it from the data. Therefore, in HSIVI-SM, the score function is fixed, not optimized. However, the semi-implicit variational posterior does not have a tractable score function. To deal with it, HSIVI-SM rewrites the Fisher divergence as the maximum of an inner optimization problem. This allows us to take advantage of the hierarchical structure of semi-implicit distributions and use a similar trick to denoising score matching for a tractable training objective. This technique is borrowed from the work of SIVI-SM (https://openreview.net/forum?id=sd90a2ytrt). We want to emphasize that the improvement of sampling efficiency of HSIVI over denoising diffusion models is not due to the way score matching is done, but the expressiveness of semi-implicit distributions and a training procedure that directly matches the marginal distributions.\n\nQ4: This work only deals with the geometric ... \n\nA4: In addition to the geometric interpolation (Example 1), we also introduced an alternative interpolation method called diffusion bridge (Example 2) which is useful for diffusion models. We also give a comparison of these interpolations in Figure 2 (geometric interpolation results) and Figure 7 in Appendix D.1 (diffusion bridge results) on the Gaussian mixture model. Exploring the impact of the choice of interpolation is interesting. However, for diffusion models, this would also be a bit challenging as it relates to how one constructs the forward process that allows efficient score estimation. We thank the reviewer for your suggestion, and we will leave a more thorough investigation for future work."}]

Author rebuttal:
We thank all reviewers for their constructive feedback. First, we'd like to address some common issues raised by the reviewers. 

### Novelty/Originality

As a semi-implicit variational inference method, HSIVI further enhances the expressiveness of semi-implicit variational posteriors by allowing multiple conditional layers. This makes HSIVI able to provide an accurate approximation of complicated distributions that would not be approximated well via SIVI which uses a single conditional layer. Moreover, we also introduce a sequence of auxiliary distributions for progressive training that further improves the stability of the algorithm.

For diffusion model acceleration, HSIVI-SM can produce high-quality samples that compare favorably to other baseline methods with small numbers of function evaluations. We want to emphasize that it is the expressiveness of semi-implicit distributions and the variational inference approach that progressively matches a sequence of auxiliary distributions (i.e., the diffusion bridge given by the learned score nets) that contributes to the improved sampling efficiency of HSIVI-SM. Although DDPM is originally derived via a variational approach, it matches the joint distributions instead of the marginal distributions. In HSIVI-SM, the marginal distribution of the backward models (i.e., semi-implicit distributions $q_t(x_t), t=T-1\ldots,0$) and the forward models $p_t(x_t), t=T-1,\\ldots,0$ are directly matched via score matching, and we see this as one of the keys that allow HSIVI-SM to generate high-quality samples with a small function evaluations (see the discussion paragraph in Section 3.2, i.e. line 150-157, for more details). Note that this also allows a direct compression of stochastic diffusion models, which is different from most of the current ODE-based acceleration methods. HSIVI-SM can also be trained without data, which would be useful when there are privacy concerns.

Overall, we think there are significant methodological innovations in HSIVI that make it a novel/original contribution to the community.

### More experiments

We have provided additional results on CelebA (64x64) and ImageNet (64x64) with more powerful pre-trained score nets (bigger models with more parameters). These results show that our methods work well for common benchmark datasets for diffusion models and can benefit from more accurate score function estimation.

Table: Sample quality measured by FID on CelebA (64x64). All methods without \* employ the same UNet in [1].
| Model\NFE | 5  | 10 | 15 |
| ---- | ---- | ---- | ---- | 
| DDPM  | 366.10 | 309.95 | 206.92 |
| DDIM  | 27.38 | 10.89 | 7.78 |
| FastDPM  | 27.63 | 15.44 | 12.05 |
| Analytic-DDPM  | 50.92 | 28.93 | 21.84 |
| Analytic-DDIM | 29.40 |  15.74 | 12.25 |
| DPM-Solver-fast | 355.96 |  6.76 | 2.98 |
| **HSIVI-SM (ours), in the main text (smaller UNet)\*** | 8.29 |  4.95 | 4.66 |
| **HSIVI-SM (ours)** | **6.22** | **3.09** | **2.23** |

Table: Sample quality measured by FID on ImageNet (64x64). All methods employ the same UNet in [2].
| Model\NFE | 5  | 10 | 15 |
| ---- | ---- | ---- | ---- | 
| DDPM  | 402.68 | 358.80 | 284.00 |
| DDIM  | 147.03 | 42.31 | 24.85 |
| Analytic-DDPM  | N/A | 60.65 | 45.98 |
| Analytic-DDIM | N/A |  70.62 | 41.56 |
| DPM-Solver-fast | 402.43 | 28.96 | 20.03 |
| **HSIVI-SM (ours)** | **40.43** | **17.67** | **15.49** |

We will revise our manuscript in the following aspects:

- We will add these additional results on CelebA (64x64), ImageNet (64x64) to the experiments.


- We will enhance the discussion of HSIVI's computational complexity and memory usage in comparison to the baseline methods.
- We will add the other relevant baseline methods (e.g., Diffusion Normalizing Flows, Rectified Flow) to the experiments and the discussion.
- We will add the ablation studies to discuss:
  - The failure case of HSIVI when the number of layers is small.
  - The effect of the accuracy of score net.
- We will add more discussion on the detaching operation and its effect on the optimality of $\phi$ in Appendix C.3.  

[1] Ho, Jonathan, Ajay Jain, and Pieter Abbeel. "Denoising diffusion probabilistic models." in NeurIPS 2020.\
[2] Nichol, Alexander Quinn, and Prafulla Dhariwal. "Improved denoising diffusion probabilistic models." in ICML 2021.

We hope our revision has adequately addressed the reviewers' questions and concerns, and look forward to reading any further comments.

Meta-review: Six reviewers evaluated this submission and the scores are 676663. All reviewers agree that this paper proposes a novel or interesting approach. Some concerns on experimental setup and complexity of practical implementation have been raised and adequately addressed in the rebuttal. The only reviewer who gave negative scores had lowered their confidence score after discussion. Therefore, I recommend acceptance.
Decision: Accept (poster)