SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Longlin  $\mathbf{Y}\mathbf{u}^{1,*}$  , Tianyu Xie $^{1,*}$ , Yu Zhu $^{3,4,*}$ , Tong Yang $^{5}$ , Xiangyu Zhang $^{5}$ , Cheng Zhang $^{1,2,\dagger}$
$^{2}$  Center for Statistical Science, Peking University
$^{3}$  Institute of Automation, Chinese Academy of Sciences
<sup>4</sup> Beijing Academy of Artificial Intelligence
5 Megvii Technology Inc.
{llyu, tianyuxie}@pku.edu.cn, zhuyu2022@ia.ac.cn,
{yangtong, zhangxiangyu}@megvii.com, chengzhang@math.pku.edu.cn
Semi-implicit variational inference (SIVI) has been introduced to expand the analytical variational families by defining expressive semi-implicit distributions in a hierarchical manner. However, the single-layer architecture commonly used in current SIVI methods can be insufficient when the target posterior has complicated structures. In this paper, we propose hierarchical semi-implicit variational inference, called HSIVI, which generalizes SIVI to allow more expressive multi-layer construction of semi-implicit distributions. By introducing auxiliary distributions that interpolate between a simple base distribution and the target distribution, the conditional layers can be trained by progressively matching these auxiliary distributions one layer after another. Moreover, given pre-trained score networks, HSIVI can be used to accelerate the sampling process of diffusion models with the score matching objective. We show that HSIVI significantly enhances the expressiveness of SIVI on several Bayesian inference problems with complicated target distributions. When used for diffusion model acceleration, we show that HSIVI can produce high quality samples comparable to or better than the existing fast diffusion model based samplers with a small number of function evaluations on various datasets.
Variational inference (VI) is an approximate Bayesian inference method that is gaining in popularity, where one tries to find an approximation to the target posterior distribution using an optimization approach (Jordan et al., 1999; Wainwright & Jordan, 2008; Blei et al., 2016). To do that, it first posits a family of variational distributions and then seeks the closest member from this family that minimizes some statistical distance to the target posterior, usually the Kullback-Leibler (KL) divergence. As the posterior is not analytically available, an equivalent formulation is often adopted in practice where one maximizes the evidence lower bound (ELBO) instead (Jordan et al., 1999).
One classical VI method is mean-field VI, which assumes a factorizable structure of the variational distributions over the parameters or latent variables (Bishop & Tipping, 2000). This often leads to closed-form coordinate-ascent update rules when certain conditional conjugacy conditions are satisfied. In practice, the conditional conjugacy may not hold and the true posterior could be much
*Equal contribution. This work was done during an internship at Megvii Technology Inc.
more complicated than what a factorized variational distribution can accurately approximate. In recent years, several attempts have been made in VI that alleviate these constraints by designing more flexible variational families (Jaakkola & Jordan, 1998; Saul & Jordan, 1996; Giordano et al., 2015; Tran et al., 2015; Rezende & Mohamed, 2015; Dinh et al., 2017; Kingma et al., 2016; Papamakarios et al., 2019), together with generic training algorithms via Monte Carlo gradient estimators (Nott et al., 2012; Paisley et al., 2012; Ranganath et al., 2014; Rezende et al., 2014; Kingma & Welling, 2014). While successful, these approaches all assume tractable densities of variational distributions. To further expand the capacity of variational families, one approach is to incorporate the implicit models that have intractable densities but are easy to sample from (Huszár, 2017; Tran et al., 2017; Mescheder et al., 2017; Shi et al., 2018a,b; Song et al., 2019). However, as the densities are intractable for implicit models, one often resorts to density ratio estimation for ELBO evaluation during training, which is known to be difficult in high dimensional settings (Sugiyama et al., 2012). To avoid density ratio estimation, semi-implicit variational inference (SIVI) has been proposed where the variational distributions are formed through a semi-implicit hierarchical construction, and various training criteria have been employed (Yin & Zhou, 2018; Moens et al., 2021; Titsias & Ruiz, 2019; Yu & Zhang, 2023).
While striking a good balance between approximation flexibility and training efficiency, current SIVI methods often use a single conditional layer which can be insufficient when the target posterior possesses complicated structures (e.g., multimodality, see an example in Section 5.1). To enhance the expressiveness of single-layer models, an intuitive but effective approach is to extend them to multi-layer hierarchical models (Vahdat & Kautz, 2020; Ranganath et al., 2016; Sobolev & Vetrov, 2019). In this paper, we propose hierarchical semi-implicit variational inference (HSIVI), which is a generalization of SIVI that allows multiple conditional layers. Instead of training the hierarchical semi-implicit model end to end, we introduce auxiliary distributions that interpolate between a simple base distribution and the target distribution to guide the intermediate semi-implicit distributions toward the target distribution. The conditional layers are then trained sequentially to match these auxiliary bridging distributions given the fitted semi-implicit distributions from the previous layers (Figure 1), using different criteria from before. This way, HSIVI allows progressive learning of the target distribution that significantly reduces the burden of each conditional layer. Moreover, HSIVI with the score matching objective can also be used to accelerate the sampling process of diffusion models where the pre-trained score networks corresponding to different noise levels provide a natural sequence of bridging distributions. In experiments, we demonstrate the effectiveness of HSIVI on both Bayesian inference tasks with complicated target distributions and diffusion model acceleration.
2 Background on semi-implicit variational inference
The semi-implicit variational family (Yin & Zhou, 2018; Titsias & Ruiz, 2019) is defined as
$$
q _ {\phi} (\boldsymbol {x}) = \int q _ {\phi} (\boldsymbol {x} | \boldsymbol {z}) q (\boldsymbol {z}) \mathrm {d} \boldsymbol {z}, \tag {1}
$$
where  $\phi$  are the variational parameters,  $q_{\phi}(\pmb{x}|\pmb{z})$  is called the conditional layer, and  $q(z)$  is called the mixing layer. This variational family is said to be semi-implicit as  $q_{\phi}(\pmb{x}|\pmb{z})$  is required to be explicit and  $q(z)$  is often implicit. The semi-implicit variational family is capable of capturing more complicated dependencies between variables (Yin & Zhou, 2018; Titsias & Ruiz, 2019; Yu & Zhang, 2023) than explicit variational families without the hierarchical structure. Given the observed data  $D$ , the classical VI methods often use the evidence lower bound (ELBO) for training, which is defined as ELBO :=  $\mathbb{E}_{q_{\phi}(\pmb{x})}\left[\log p(D,\pmb{x}) - \log q_{\phi}(\pmb{x})\right]$ . However, as  $q_{\phi}(\pmb{x})$  is no longer tractable in SIVI, alternative training objectives have been introduced.
ELBO related objectives Yin & Zhou (2018) considered a sequence of lower bounds of the ELBO
$$
\mathcal {L} _ {\text {S I V I - L B}} \left(p (\boldsymbol {x} | D) \| q _ {\phi} (\boldsymbol {x})\right) := \mathbb {E} _ {\boldsymbol {z} \sim q (\boldsymbol {z}), \boldsymbol {x} \sim q _ {\phi} (\boldsymbol {x}, \boldsymbol {z})} \mathbb {E} _ {\left\{\boldsymbol {z} ^ {(i)} \right\} _ {i = 1} ^ {K} \stackrel {{\text {i . i . d .}}} {{\sim}} q (\boldsymbol {z})} \log \frac {p (D , \boldsymbol {x})}{\frac {1}{K + 1} \left(q _ {\phi} (\boldsymbol {x} | \boldsymbol {z}) + \sum_ {k = 1} ^ {K} q _ {\phi} (\boldsymbol {x} | \boldsymbol {z} ^ {(k)})\right)}. \tag {2}
$$
It is an asymptotically exact surrogate in the sense that  $\lim_{K\to \infty}\mathcal{L}_{\mathrm{SIVI - LB}} = \mathrm{ELBO}$ . Titsias & Ruiz (2019) proposed unbiased implicit variational inference (UIVI) which uses samples from the inverse conditional distribution  $q_{\phi}(\pmb {z}|\pmb {x})$  (from an MCMC run, e.g. Hamiltonian Monte Carlo (Neal, 2011)) to provide an unbiased gradient estimator of the exact ELBO. See more details of UIVI in Appendix B.
Figure 1: An example for 4-layer HSIVI. The target distribution  $p_0(\pmb{x})$  is a Gaussian mixture and the auxiliary distributions  $\{p_i(\pmb{x})\}_{i=0}^3$  are constructed using the diffusion bridge. The auxiliary distributions are plotted in the squares, where the blue heatmap describes the probability density and the arrows represent the score functions of the auxiliary distributions.
Score matching objective Besides the ELBO, score based distance measures have also been used for variational inference where the score function  $S(\pmb{x}) \coloneqq \nabla_{\pmb{x}} \log p(\pmb{x} | D) = \nabla_{\pmb{x}} \log p(D, \pmb{x})$  is assumed to be tractable (Liu et al., 2016; Zhang et al., 2018; Hu et al., 2018). Yu & Zhang (2023) considered the following Fisher divergence between the target distribution and the semi-implicit variational distribution
$$
\mathcal {D} _ {\text {F i s h e r}} \left(p (\boldsymbol {x} | D) \| q _ {\phi} (\boldsymbol {x})\right) := \mathbb {E} _ {\boldsymbol {x} \sim q _ {\phi} (\boldsymbol {x})} \| \boldsymbol {S} (\boldsymbol {x}) - \nabla_ {\boldsymbol {x}} \log q _ {\phi} (\boldsymbol {x}) \| _ {2} ^ {2}. \tag {3}
$$
By reformulating  $\mathcal{D}_{\mathrm{Fisher}}$  as the maximum of the following optimization problem
$$
\mathcal {D} _ {\mathrm {F i s h e r}} (p (\boldsymbol {x} | D) \| q _ {\phi} (\boldsymbol {x})) = \max _ {\boldsymbol {f} (\boldsymbol {x})} \left[ 2 \boldsymbol {f} (\boldsymbol {x}) ^ {T} (\boldsymbol {S} (\boldsymbol {x}) - \nabla_ {\boldsymbol {x}} \log q _ {\phi} (\boldsymbol {x})) - \| \boldsymbol {f} (\boldsymbol {x}) \| _ {2} ^ {2} \right],
$$
and using a similar trick as in denoising score matching (Vincent, 2011; Song & Ermon, 2019), one can transform the minimization of  $\mathcal{D}_{\mathrm{Fisher}}$  into the following minimax problem which is tractable
$$
\min  _ {\phi} \max  _ {\psi} \mathcal {L} _ {\mathrm {S I V I - S M}} (p (\boldsymbol {x} | D) \| q _ {\phi} (\boldsymbol {x})) := \mathbb {E} _ {\boldsymbol {z} \sim q (\boldsymbol {z}), \boldsymbol {x} \sim q _ {\phi} (\boldsymbol {x} | \boldsymbol {z})} \left[ 2 \boldsymbol {f} _ {\psi} (\boldsymbol {x}) ^ {T} [ \boldsymbol {S} (\boldsymbol {x}) - \nabla_ {\boldsymbol {x}} \log q _ {\phi} (\boldsymbol {x} | \boldsymbol {z}) ] - \| \boldsymbol {f} _ {\psi} (\boldsymbol {x}) \| _ {2} ^ {2} \right. \tag {4}
$$
In practice,  $f_{\psi}(\pmb{x})$  is parametrized using neural networks. The above minimax optimization problem can be efficiently solved by optimizing  $\psi$  and  $\phi$  alternately.
3 Hierarchical semi-implicit variational inference
The semi-implicit variational family  $q_{\phi}(\pmb{x})$  in equation (1) is indeed a single-layer model in the sense that it contains only one conditional layer. Our main idea is to expand this single-layer semi-implicit variational family into its multi-layer variants and introduce a sequence of auxiliary distributions to guide the semi-implicit distributions toward the target distribution. This leads to a new SIVI method which we call hierarchical semi-implicit variational inference (HSIVI). We start with the following definition which is motivated by equation (1).
Definition 1 (Hierarchical Semi-Implicit Distribution). Let  $\pmb{x}_T \sim q_T(\pmb{x}_T)$  for some  $T \in \mathbb{N}^*$ , where  $q_{T}(\pmb{x}_{T})$  is called the variational prior. Let  $q_{t}(\pmb{x}_{t}|\pmb{x}_{t + 1};\phi_{t})$  be the  $t$ -th conditional layer for  $0 \leq t \leq T - 1$ . Denote  $\{\phi_k\}_{k = t}^{T - 1}$  by  $\phi_{\geq t}$ . The  $t$ -th layer hierarchical semi-implicit distribution  $q_{t}(\pmb{x}_{t};\phi_{\geq t})$  is defined recursively from  $T - 1$  to 0 by
$$
q _ {t} \left(\boldsymbol {x} _ {t}; \phi_ {\geq t}\right) = \int q _ {t} \left(\boldsymbol {x} _ {t} \mid \boldsymbol {x} _ {t + 1}; \phi_ {t}\right) q _ {t + 1} \left(\boldsymbol {x} _ {t + 1}; \phi_ {\geq t + 1}\right) \mathrm {d} \boldsymbol {x} _ {t + 1}, \quad 0 \leq t \leq T - 1, \tag {5}
$$
where  $q_{T}(\pmb{x}_{T};\phi_{\geq T}) \coloneqq q_{T}(\pmb{x}_{T})$ . Here, the t-th conditional layer  $q_{t}(\pmb{x}_{t}|\pmb{x}_{t + 1};\phi_{t})$  is required to be explicit and reparametrizable with a tractable score function  $\nabla_{\pmb{x}_t}\log q_t(\pmb{x}_t|\pmb{x}_{t + 1};\phi_t)$ .
Compared to the single-layer semi-implicit variational family (1), the family of hierarchical semi-implicit distributions provides a principled way to construct more expressive mixing layers using multi-layer architectures. Also, unlike the hierarchical variational models (Ranganath et al., 2016) which require an extra reverse model and explicit variational prior, hierarchical semi-implicit distributions inherit the advantage of SIVI that allows  $q_{t}(\boldsymbol{x}_{t};\phi_{\geq t})$  to be implicit, and as shown next, they do not require a reverse model and can be progressively trained using the simple algorithms of SIVI for each conditional layer, from  $t = T - 1$  to  $t = 0$ .
Algorithm 1 Hierarchical semi-implicit variational inference (sequential training)
Input: Auxiliary bridge  $\{p_t(\boldsymbol {x})\}_{t = 0}^{T - 1}$  ; initial value of parameters  $\phi^{(0)} = \{\phi_i^{(0)}\}_{t = 0}^{T - 1}$    
Output: The optimal parameters  $\phi^*$    
Initialization:  $\phi \gets \phi^{(0)}$    
for  $t = T - 1$  to 0 do while not converge do Sample a minibatch  $\{\pmb{x}_T^{(k)}\}_{k = 1}^K$  from the variational prior  $q_{T}(\pmb{x}_{T})$  if  $t <   T - 1$  then Sequentially sample  $\{\pmb{x}_{t + 1}^{(k)}\}_{k = 1}^{K}$  through  $q(\pmb {x}_i|\pmb {x}_{i + 1};\phi_i)$  from  $i = T - 1$  to  $i = t + 1$  Detach the computation graphs from  $\{\pmb{x}_{t + 1}^{(k)}\}_{k = 1}^{K}$  end if Update  $\phi_t$  by optimizing the  $\mathcal{L}_{\mathrm{SIVI - f}}(p_t(\pmb {x}_t)\| q_t(\pmb {x}_t;\phi_\geq t))$  based on the minibatch  $\{\pmb{x}_{t + 1}^{(k)}\}_{k = 1}^{K}$  end while  $\phi_t^* \gets \phi_t$    
end for   
 $\phi^* \leftarrow \{\phi_t^*\}_{t = 0}^{T - 1}$
3.1 Progressive approximation with the auxiliary bridge
In this section, we introduce a bridging technique for progressively approximating the target distribution  $p(\pmb{x})$  using hierarchical semi-implicit distributions. Rather than approximating  $p(\pmb{x})$  with  $q_0(\pmb{x};\phi_{\geq 0})$  directly, we construct a sequence of intermediate auxiliary distributions  $\{p_t(\pmb{x})\}_{t=0}^{T-1}$  as a bridge between the target distribution  $p_0(\pmb{x}) := p(\pmb{x})$  and an easy-to-approximate distribution  $p_{T-1}(\pmb{x})$ , to amortize the difficulty of one-pass fitting. A typical example of an auxiliary bridge is the geometric interpolation as described below.
Example 1 (Geometric Interpolation). Let  $S(\pmb{x}) \coloneqq \nabla \log p(\pmb{x})$  be the score function of target distribution  $p(\pmb{x})$  and  $S_{base}(\pmb{x}) \coloneqq \nabla \log p_{base}(\pmb{x})$  be the score function of a base distribution  $p_{base}(\pmb{x})$ . In geometric interpolation (Neal, 2001; Bernton et al., 2019), each auxiliary distribution  $p_t(\pmb{x})$  for  $0 \leq t \leq T - 1$  has the following probability density function (pdf) and score function
$$
p _ {t} (\boldsymbol {x}) \propto p _ {\text {b a s e}} (\boldsymbol {x}) ^ {1 - \lambda_ {t}} p (\boldsymbol {x}) ^ {\lambda_ {t}}, \quad \boldsymbol {S} _ {t} (\boldsymbol {x}) := \nabla_ {\boldsymbol {x}} \log p _ {t} (\boldsymbol {x}) = (1 - \lambda_ {t}) \boldsymbol {S} _ {\text {b a s e}} (\boldsymbol {x}) + \lambda_ {t} \boldsymbol {S} (\boldsymbol {x}), \tag {6}
$$
where  $\{\lambda_t\}_{t=0}^{T-1}$  is a non-negative decreasing sequence satisfying  $\lambda_0 = 1$ .
Intuitively, we expect the distance between two neighboring distributions  $p_t(\pmb{x})$  and  $p_{t+1}(\pmb{x})$  to be not too large so that it would be easy to construct a conditional distribution  $q_t(\pmb{x}_t | \pmb{x}_{t+1})$  such that  $p_t(\pmb{x}_t) \approx \int q_t(\pmb{x}_t | \pmb{x}_{t+1}) p_{t+1}(\pmb{x}_{t+1}) \mathrm{d}\pmb{x}_{t+1}$ . Note that the auxiliary bridge  $\{p_t(\pmb{x})\}_{t=0}^{T-1}$  does not necessarily need to have analytical pdfs (up to a constant). In fact, it suffices if they have tractable score functions  $\{S_t(\pmb{x})\}_{t=0}^{T-1}$  which lead to another type of auxiliary bridge (Example 2 in Section 4).
3.2 Sequential training of HSIVI
Given the auxiliary distributions  $\{p_t(\boldsymbol{x}_t)\}_{t=0}^{T-1}$ , a natural approach is to progressively train the hierarchical semi-implicit distribution  $q_t(\boldsymbol{x}_t; \phi_{\geq t})$  to match  $p_t(\boldsymbol{x}_t)$  from  $t = T - 1$  to  $t = 0$ . Let the parameters  $\phi_t$  in the  $t$ -th conditional layer be independent across different  $ts$ . We first train  $q_{T-1}(\boldsymbol{x}_{T-1}; \phi_{T-1})$  to match  $p_{T-1}(\boldsymbol{x}_{T-1})$  by optimizing  $\phi_{T-1}$  w.r.t. the single-layer SIVI objective  $\mathcal{L}_{\mathrm{SIVI}-f}(p_{T-1}(\boldsymbol{x}_{T-1}) \| q_{T-1}(\boldsymbol{x}_{T-1}; \phi_{T-1}))$ . For  $t = T - 2, \ldots, 0$ , given the trained semi-implicit distribution  $q_{t+1}(\boldsymbol{x}_{t+1}; \phi_{\geq t+1})$ , we can fix it as the mixing layer and train the  $t$ -th conditional layer  $q_t(\boldsymbol{x}_t | \boldsymbol{x}_{t+1}; \phi_t)$  by optimizing  $\phi_t$  w.r.t. the single-layer SIVI objective  $\mathcal{L}_{\mathrm{SIVI}-f}(p_t(\boldsymbol{x}_t) \| q_t(\boldsymbol{x}_t; \phi_{\geq t}))$  as well. Note this is fine as the mixing layer can be implicit in SIVI. Here,  $f$  is some distance criterion, e.g.,  $\mathcal{L}_{\mathrm{SIVI-LB}}$  in equation (2) or  $\mathcal{L}_{\mathrm{SIVI-SM}}$  in equation (4). In this article, we mainly focus on  $\mathcal{L}_{\mathrm{SIVI-LB}}$  and  $\mathcal{L}_{\mathrm{SIVI-SM}}$ , while other distance criteria can also be applied. We summarize this sequential training procedure in Algorithm 1.
Score based training In addition to the common assumption that  $p_t(\pmb{x})$  is known up to a constant, it is worth noting that  $\mathcal{L}_{\mathrm{SIVI-LB}}$  is also applicable when only the score functions  $\{S_t(\pmb{x})\}_{t=0}^{T-1}$  are available which is important for the diffusion bridge construction of auxiliary distributions in Example 2. Concretely, assume  $q_t(\pmb{x}_t | \pmb{x}_{t+1}; \phi_t)$  is induced by a parametrized transform  $\pmb{x}_t = \pmb{h}_t(\pmb{x}_{t+1}, \pmb{\epsilon}; \phi_t)$ .
where  $\epsilon \sim p_{\epsilon}(\epsilon)$  is a random noise. The only term in  $\mathcal{L}_{\mathrm{SIVI-LB}}(p_t(\boldsymbol{x}_t) \| q_t(\boldsymbol{x}_t; \phi_{\geq t}))$  containing  $p_t(\boldsymbol{x}_t)$  is  $\mathbb{E}_{q_t(\boldsymbol{x}_t; \phi_{\geq t})} \log p_t(\boldsymbol{x}_t)$  (see equation (2)) whose gradient takes the form
$$
\nabla_ {\phi_ {t}} \mathbb {E} _ {q _ {t} (\boldsymbol {x} _ {t}; \phi_ {\geq t})} \log p _ {t} (\boldsymbol {x} _ {t}) = \mathbb {E} _ {q _ {t + 1} (\boldsymbol {x} _ {t + 1}; \phi_ {\geq t + 1}) p _ {\epsilon} (\boldsymbol {\epsilon})} S _ {t} \left(\boldsymbol {h} _ {t} (\boldsymbol {x} _ {t + 1}, \boldsymbol {\epsilon}; \phi_ {t})\right) \nabla_ {\phi_ {t}} \boldsymbol {h} _ {t} (\boldsymbol {x} _ {t + 1}, \boldsymbol {\epsilon}; \phi_ {t}). \tag {7}
$$
In the training of HSIVI-SM, each term  $\mathcal{L}_{\mathrm{SIVI - SM}}(p_t(\boldsymbol{x}_t)\| q_t(\boldsymbol{x}_t;\phi_{\geq t}))$  involves a nested optimization of  $f_{t}(\boldsymbol{x}_{t};\psi_{t})$ . When the score functions are computationally expensive, we find that an alternative parametrization  $f_{t}(\boldsymbol{x}_{t};\psi_{t}):= S_{t}(\boldsymbol{x}_{t}) - g_{t}(\boldsymbol{x}_{t};\psi_{t})$  is useful to avoid the time-consuming evaluation of  $S_{t}(\boldsymbol{x}_{t})$  when optimizing  $\psi_t$  in equation (4). The reason for this lies in Proposition 1. See Appendix C.2 for the proof of Proposition 1.
Proposition 1. Let  $q_{t}(\pmb{x}_{t}, \pmb{x}_{t+1}; \phi_{\geq t}) = q_{t}(\pmb{x}_{t} | \pmb{x}_{t+1}; \phi_{t}) q_{t+1}(\pmb{x}_{t+1}; \phi_{\geq t+1})$ . The minimax optimization of  $\mathcal{L}_{SIVI-SM}(p_{t}(\pmb{x}_{t}) \| q_{t}(\pmb{x}_{t}; \phi_{\geq t}))$  is equivalent to
$$
\min _ {\boldsymbol {\phi} _ {t}} \mathbb {E} _ {q _ {t} (\boldsymbol {x} _ {t}, \boldsymbol {x} _ {t + 1}; \boldsymbol {\phi} _ {\ge t})} \left[ \boldsymbol {S} _ {t} (\boldsymbol {x} _ {t}) - \boldsymbol {g} _ {t} (\boldsymbol {x} _ {t}; \boldsymbol {\psi} _ {t}) \right] ^ {T} \left[ \boldsymbol {S} _ {t} (\boldsymbol {x} _ {t}) + \boldsymbol {g} _ {t} (\boldsymbol {x} _ {t}; \boldsymbol {\psi} _ {t}) - 2 \nabla_ {\boldsymbol {x} _ {t}} \log q _ {t} (\boldsymbol {x} _ {t} | \boldsymbol {x} _ {t + 1}; \boldsymbol {\phi} _ {t}) \right],
$$
$$
\min  _ {\boldsymbol {\psi} _ {t}} \quad \mathbb {E} _ {q _ {t} (\boldsymbol {x} _ {t}, \boldsymbol {x} _ {t + 1}; \phi_ {\geq t})} \| \boldsymbol {g} _ {t} (\boldsymbol {x} _ {t}; \psi_ {t}) - \nabla_ {\boldsymbol {x} _ {t}} \log q _ {t} (\boldsymbol {x} _ {t} | \boldsymbol {x} _ {t + 1}; \phi_ {t}) \| _ {2} ^ {2}.
$$
Marginal approximation v.s. joint approximation Previous works (Bernton et al., 2019; Bao et al., 2022) often construct a joint distribution  $p(\pmb{x}_{0:T})$  and minimize  $\mathrm{KL}(p(\pmb{x}_{0:T-1})\|q(\pmb{x}_{0:T-1}))$  where  $q(\pmb{x}_{0:T-1})$  is a variational distribution. In HSIVI, we directly approximate  $p_t(\pmb{x}_t)$  using the semi-implicit variational distributions. When  $p(\pmb{x}_{0:T-1})$  is complex and  $T$  is small, the variational distribution  $q(\pmb{x}_{0:T-1})$  may be insufficient to fully capture the joint distribution  $p(\pmb{x}_{0:T-1})$ . For example, the optimal fit of the joint distribution for diffusion models established by Analytic-DPM (Bao et al., 2022) does not guarantee that the marginal distributions would be approximated well (see Table 2 for comparison).
4 Application to diffusion model acceleration
4.1 Review of diffusion models
Recently, diffusion models have shown great success on many generative modeling benchmarks, including image generation (Ho et al., 2020; Song et al., 2020a,b), graph generation (Niu et al., 2020), and text generation (Austin et al., 2021). Diffusion models work by adding noise to the training data in the forward process and then removing the noise to recover the data in the backward process, which can be integrated into a general stochastic differential equation (SDE) framework. The forward process  $\{\pmb{u}_s\}_{s\in [0,L]}$  is usually described by
$$
\mathrm {d} \boldsymbol {u} _ {s} = \boldsymbol {f} (\boldsymbol {u} _ {s}, s) \mathrm {d} s + g (s) \mathrm {d} \boldsymbol {w} _ {s}, \quad \boldsymbol {u} _ {0} \sim p _ {0} (\cdot), \tag {8}
$$
where  $p_0(\cdot)$  is the data distribution,  $\boldsymbol{w}_s$  is a standard Brownian motion, and  $\boldsymbol{f}(\boldsymbol{u}_s, s)$  and  $g(s)$  are the drift and diffusion coefficients respectively. To generate samples from the data distribution, one can run the following backward process
$$
\mathrm {d} \boldsymbol {u} _ {s} = [ \boldsymbol {f} (\boldsymbol {u} _ {s}, s) - g ^ {2} (s) \nabla_ {\boldsymbol {u} _ {s}} \log p _ {s} (\boldsymbol {u} _ {s}) ] \mathrm {d} s + g (s) \mathrm {d} \bar {\boldsymbol {w}} _ {s}, \quad \boldsymbol {u} _ {L} \sim p _ {L} (\cdot), \tag {9}
$$
where  $p_{s}(\cdot)$  is the pdf of  $\mathbf{u}_s$  and  $\bar{\boldsymbol{w}}_s$  is a standard Brownian motion when time flows from  $L$  to 0. As the score function  $\nabla_{\mathbf{u}_s}\log p_s(\mathbf{u}_s)$  is intractable, we need to estimate it by denoising score matching (Vincent, 2011; Song et al., 2020b). See more details of diffusion models and the training objectives in Appendix A.
4.2 Diffusion model acceleration via HSIVI