SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In practice, Song et al. (2020b) and Kingma et al. (2021) designed several examples of the forward process such that it diffuses the data distribution  $p_0(\cdot)$  to a fixed unstructured distribution  $p_L(\cdot)$ . Here we mainly consider the Variance Preserving SDE (VP-SDE) used in DDPM (Ho et al., 2020; Song et al., 2020b). Let the drift coefficient  $f(\boldsymbol{u}_s, s) = \frac{\mathrm{d}\log\alpha(s)}{2\mathrm{d}s} \boldsymbol{u}_s$  and the diffusion coefficient  $g^2(s) = -\frac{\mathrm{d}\log\alpha(s)}{\mathrm{d}s}$ , where  $\alpha(s) \in \mathbb{R}^+$  is a decreasing smooth function with  $\alpha(0) = 1$ ,  $\alpha(L) \approx 0$ . Then the distribution of  $\boldsymbol{u}_s$  conditioned on  $\boldsymbol{u}_0$  is explicit as
$$
\boldsymbol {u} _ {s} \left| \boldsymbol {u} _ {0} \sim \mathcal {N} \left(\sqrt {\alpha (s)} \bar {\boldsymbol {x}} _ {0}, (1 - \alpha (s)) \mathbf {I}\right), \text {i . e .} \boldsymbol {u} _ {s} = \sqrt {\alpha (s)} \boldsymbol {u} _ {0} + \sqrt {1 - \alpha (s)} \boldsymbol {\epsilon}, \right. \tag {12}
$$
where  $\epsilon$  is a standard Gaussian noise. In practice, diffusion models use a neural network  $S_{\theta}(\pmb{u}_s,s)$  to approximate the score function  $S_{\theta}(\pmb{u}_s,s)$  by optimizing the denoising score matching objective (Vincent, 2011)
$$
\mathcal {L} _ {\mathrm {d s m}} (\theta , \omega (s)) := \frac {1}{2} \int_ {0} ^ {L} \omega (s) \mathbb {E} _ {\boldsymbol {u} _ {0} \sim p _ {0} (\boldsymbol {u} _ {0}), \epsilon \sim \mathcal {N} (0, \mathbf {I})} \| \boldsymbol {S} _ {\theta} (\boldsymbol {u} _ {s}, s) + \epsilon / \sqrt {1 - \alpha (s)} \| _ {2} ^ {2} \mathrm {d} s, \tag {13}
$$
where  $\omega(s)$  is a positive weighting function. Instead of modeling the score function, Ho et al. (2020) proposed to predict the conditional noise  $\epsilon$  based on  $\pmb{u}_t$ . This leads to the following DDPM loss
$$
\mathcal {L} _ {\mathrm {d d p m}} (\theta , \bar {\omega} (s)) := \frac {1}{2} \int_ {0} ^ {L} \bar {\omega} (s) \mathbb {E} _ {\boldsymbol {u} _ {0} \sim p _ {0} (\boldsymbol {u} _ {0}), \epsilon \sim \mathcal {N} (0, \mathbf {I})} \| \epsilon_ {\theta} (\boldsymbol {u} _ {s}, s) - \epsilon \| _ {2} ^ {2} \mathrm {d} s, \tag {14}
$$
where  $\bar{\omega}(s)$  is a positive weighting function. In fact, we have the relationship
$$
\boldsymbol {S} _ {\theta} \left(\boldsymbol {u} _ {s}, s\right) = - \epsilon_ {\theta} \left(\boldsymbol {u} _ {s}, s\right) / \sqrt {1 - \alpha (s)}. \tag {15}
$$
We call  $\mathcal{L}_{\mathrm{dsm}}$  "score-prediction" training and  $\mathcal{L}_{\mathrm{ddpm}}$  "ε-prediction" training.
With the pre-trained score model  $S_{\theta}(\boldsymbol{u}_s, s)$  or noise model  $\epsilon_{\theta}(\boldsymbol{u}_s, s)$ , Song et al. (2020b) shows that the samples of  $p_0(\cdot)$  can be generated by simulating the backward SDE, e.g. the sampling scheme of DDPM (Ho et al., 2020). Moreover, Bao et al. (2022) proposed Analytic-DPM, the optimal discretization form responding to the KL divergence of the joint distribution on the discrete time steps. Also, several high-order ODE solvers (Song et al., 2020a; Zhang & Chen, 2022; Lu et al., 2022) were proposed to achieve faster sampling.
B More details of UIVI
Unlike optimizing the surrogate ELBO, Titsias & Ruiz (2019) proposed unbiased implicit variational inference (UIVI) which relies on an unbiased gradient estimator for the exact ELBO. To elaborate further, reparametrize the conditional  $q_{\phi}(\pmb{x}|\pmb{z})$  such as  $\pmb{x} = T_{\phi}(\pmb{z},\epsilon), \epsilon \sim q_{\epsilon}(\epsilon)$ , then
$$
\begin{array}{l} \nabla_ {\phi} \operatorname {E L B O} = \nabla_ {\phi} \mathbb {E} _ {\epsilon \sim q _ {\epsilon} (\epsilon), z \sim q (z)} \left[ \log p (D, \boldsymbol {x}) - \log q _ {\phi} (\boldsymbol {x}) | _ {\boldsymbol {x} = T _ {\phi} (\boldsymbol {z}, \epsilon)} \right] \\ = \mathbb {E} _ {\boldsymbol {\epsilon} \sim q _ {\boldsymbol {\epsilon}} (\boldsymbol {\epsilon}), \boldsymbol {z} \sim q (\boldsymbol {z})} \left[ g _ {\phi} ^ {\mathrm {m o d}} (\boldsymbol {z}, \boldsymbol {\epsilon}) + g _ {\phi} ^ {\mathrm {e n t}} (\boldsymbol {z}, \boldsymbol {\epsilon}) \right], \\ \end{array}
$$
$$
g _ {\phi} ^ {\mathrm {m o d}} (\boldsymbol {z}, \boldsymbol {\epsilon}) := \left. \nabla_ {\boldsymbol {x}} \log p (D, \boldsymbol {x}) \right| _ {\boldsymbol {x} = T _ {\phi} (\boldsymbol {z}, \boldsymbol {\epsilon})} \nabla_ {\phi} T _ {\phi} (\boldsymbol {z}, \boldsymbol {\epsilon}),
$$
$$
\left. g _ {\phi} ^ {\text {e n t}} (\boldsymbol {z}, \epsilon) := - \mathbb {E} _ {q _ {\phi} \left(\boldsymbol {z} ^ {\prime} \mid \boldsymbol {x}\right)} \nabla_ {\boldsymbol {x}} \log q _ {\phi} \left(\boldsymbol {x} \mid \boldsymbol {z} ^ {\prime}\right) \right| _ {\boldsymbol {x} = T _ {\phi} (\boldsymbol {z}, \epsilon)} \nabla_ {\phi} T _ {\phi} (\boldsymbol {z}, \epsilon). \tag {16}
$$
The second gradient term  $g_{\phi}^{\mathrm{ent}}$  involves an expectation w.r.t. the reverse conditional  $q_{\phi}(z'|x)$  which is estimated by an MCMC sampler in UIVI. However, the inner-loop MCMC runs may require long iterations for convergence.
C More details of HSIVI
C.1 Score-based training of HSIVI-LB
In the sequential training of HSIVI-LB, although the objective  $\mathcal{L}_{\mathrm{SIVI - LB}}(p_t(\boldsymbol {x}_t)\| q_t(\boldsymbol {x}_t;\phi_{\geq t}))$  is calculated based on  $p_t(x)$ , the gradient of it w.r.t.  $\phi_t$  has a closed form containing only the score function  $S_{t}(\pmb {x})$  without knowing the corresponding pdfs. This derivation is important in the tasks where score functions of the auxiliary distributions are tractable while pdfs (up to a constant) of them are unavailable (for example, the diffusion bridge in Example 2). Concretely, assume the  $t$ -th conditional layer  $q_{t}(\pmb{x}_{t}|\pmb{x}_{t + 1};\phi_{t})$  is induced by a parametrized transform  $\pmb {x}_t = \pmb {h}_t(\pmb {x}_{t + 1},\pmb {\epsilon};\phi_t)$  where  $\epsilon \sim p_{\epsilon}(\epsilon)$  is a random noise, since  $q_{t}(\pmb{x}_{t}|\pmb{x}_{t + 1};\phi_{t})$  is reparametrizable according to Definition 1. The only term in  $\mathcal{L}_{\mathrm{SIVI - LB}}(p_t(\pmb {x}_t)\| q_t(\pmb {x}_t;\phi_{\geq t}))$  containing  $p_t(\pmb {x}_t)$  is  $\mathbb{E}_{q_t(\pmb {x}_t;\phi_{\geq t})}\log p_t(\pmb {x}_t)$  (see equation (2)) whose gradient takes the form
$$
\begin{array}{l} \nabla_ {\phi_ {t}} \mathbb {E} _ {q _ {t} (\boldsymbol {x} _ {t}; \phi_ {\ge t})} \log p _ {t} (\boldsymbol {x} _ {t}) = \nabla_ {\phi_ {t}} \mathbb {E} _ {q _ {t + 1} (\boldsymbol {x} _ {t + 1}; \phi_ {\ge t + 1}) p _ {\epsilon} (\epsilon)} \log p _ {t} (\boldsymbol {h} _ {t} (\boldsymbol {x} _ {t + 1}, \epsilon ; \phi_ {t})) \\ = \mathbb {E} _ {q _ {t + 1} (\boldsymbol {x} _ {t + 1}; \phi_ {\ge t + 1}) p _ {\epsilon} (\epsilon)} \boldsymbol {S} _ {t} \left(\boldsymbol {h} _ {t} (\boldsymbol {x} _ {t + 1}, \epsilon ; \phi_ {t})\right) \nabla_ {\phi_ {t}} \boldsymbol {h} _ {t} (\boldsymbol {x} _ {t + 1}, \epsilon ; \phi_ {t}) \\ \end{array}
$$
by the chain rule, where  $\nabla_{\phi_t}h_t(\pmb{x}_{t + 1},\pmb {\epsilon};\phi_t)$  is the jacobian matrix of  $h_t(x_{t + 1},\epsilon ;\phi_t)$ .
In our implementation of HSIVI (in both sequential training and joint training), we generally assume the conditional layer  $q_{t}(\cdot |\pmb{x}_{t + 1};\phi_t)$  is induced by
$$
\boldsymbol {h} _ {t} \left(\boldsymbol {x} _ {t + 1}, \epsilon ; \phi_ {t}\right) = \boldsymbol {\mu} _ {t} \left(\boldsymbol {x} _ {t + 1}; \phi_ {t}\right) + \boldsymbol {\Sigma} _ {t} ^ {1 / 2} \left(\boldsymbol {x} _ {t + 1}; \phi_ {t}\right) \epsilon \tag {17}
$$
where  $\pmb{\Sigma}_t(\pmb{x}_{t + 1};\phi_t)$  is a positive definite covariance matrix and  $\epsilon \sim \mathcal{N}(\mathbf{0},\mathbf{I})$  is a standard multivariate gaussian variable. In equation (17),  $\phi_t$  should be replaced by  $\phi$  in the joint training case.
C.2 Proof of Proposition 1
Proposition 1. Let  $q_{t}(\pmb{x}_{t},\pmb{x}_{t + 1};\phi_{\geq t}) = q_{t}(\pmb{x}_{t}|\pmb{x}_{t + 1};\phi_{t})q_{t + 1}(\pmb{x}_{t + 1};\phi_{\geq t + 1})$ . The minimax optimization of  $\mathcal{L}_{SIVI-SM}\left(p_t(\pmb{x}_t)\| q_t(\pmb{x}_t;\phi_{\geq t})\right)$  is equivalent to
$$
\min _ {\boldsymbol {\psi} _ {t}} \mathbb {E} _ {q _ {t} (\boldsymbol {x} _ {t}, \boldsymbol {x} _ {t + 1}; \phi_ {\ge t})} \| \boldsymbol {g} _ {t} (\boldsymbol {x} _ {t}; \boldsymbol {\psi} _ {t}) - \nabla_ {\boldsymbol {x} _ {t}} \log q _ {t} (\boldsymbol {x} _ {t} | \boldsymbol {x} _ {t + 1}; \phi_ {t}) \| _ {2} ^ {2}.
$$
Proof of Proposition 1 The minimax optimization problem of  $\mathcal{L}_{\mathrm{SIVI-SM}}(p_t(\boldsymbol{x}_t) \| q_t(\boldsymbol{x}_t; \phi_{\geq t}))$  is
$$
\min _ {\boldsymbol {\phi} _ {t}} \max _ {\boldsymbol {\psi} _ {t}} \mathbb {E} _ {q _ {t} (\boldsymbol {x} _ {t}, \boldsymbol {x} _ {t}; \boldsymbol {\phi} _ {\ge t})} \left[ 2 \boldsymbol {f} _ {t} (\boldsymbol {x} _ {t}; \boldsymbol {\psi} _ {t}) ^ {T} [ \boldsymbol {S} _ {t} (\boldsymbol {x} _ {t}) - \nabla_ {\boldsymbol {x} _ {t}} \log q _ {t} (\boldsymbol {x} _ {t} | \boldsymbol {x} _ {t + 1}; \boldsymbol {\phi} _ {t}) ] - \| \boldsymbol {f} _ {t} (\boldsymbol {x}; \boldsymbol {\psi} _ {t}) \| _ {2} ^ {2} \right]
$$
according to equation (4). For minimization w.r.t.  $\phi_t$ , this target is equivalent to
$$
\begin{array}{l} \mathbb {E} _ {q _ {t} \left(\boldsymbol {x} _ {t}, \boldsymbol {x} _ {t + 1}; \phi_ {\geq t}\right)} \left[ 2 \boldsymbol {f} _ {t} \left(\boldsymbol {x} _ {t}; \psi_ {t}\right) ^ {T} \left[ \boldsymbol {S} _ {t} \left(\boldsymbol {x} _ {t}\right) - \nabla_ {\boldsymbol {x} _ {t}} \log q _ {t} \left(\boldsymbol {x} _ {t} \mid \boldsymbol {x} _ {t + 1}; \phi_ {t}\right) \right] - \| \boldsymbol {f} _ {t} (\boldsymbol {x}; \psi_ {t}) \| _ {2} ^ {2} \right] \\ = \mathbb {E} _ {q _ {t} \left(\boldsymbol {x} _ {t}, \boldsymbol {x} _ {t + 1}; \phi_ {\geq t}\right)} \boldsymbol {f} _ {t} \left(\boldsymbol {x} _ {t}; \psi_ {t}\right) ^ {T} \left[ 2 \boldsymbol {S} _ {t} \left(\boldsymbol {x} _ {t}\right) - \boldsymbol {f} _ {t} \left(\boldsymbol {x} _ {t}; \psi_ {t}\right) - 2 \nabla_ {\boldsymbol {x} _ {t}} \log q _ {t} \left(\boldsymbol {x} _ {t} \mid \boldsymbol {x} _ {t + 1}; \phi_ {t}\right) \right] \\ = \mathbb {E} _ {q _ {t} (\boldsymbol {x} _ {t}, \boldsymbol {x} _ {t + 1}; \phi_ {\geq} t)} [ \boldsymbol {S} _ {t} (\boldsymbol {x} _ {t}) - \boldsymbol {g} _ {t} (\boldsymbol {x} _ {t}; \psi_ {t}) ] ^ {T} [ \boldsymbol {S} _ {t} (\boldsymbol {x} _ {t}) + \boldsymbol {g} _ {t} (\boldsymbol {x} _ {t}; \psi_ {t}) - 2 \nabla_ {\boldsymbol {x} _ {t}} \log q _ {t} (\boldsymbol {x} _ {t} | \boldsymbol {x} _ {t + 1}; \phi_ {t}) ]. \\ \end{array}
$$
For maximization w.r.t.  $\psi_t$ , this target is equivalent to
$$
\begin{array}{l} \mathbb {E} _ {q _ {t} \left(\boldsymbol {x} _ {t}, \boldsymbol {x} _ {t + 1}; \phi_ {\geq t}\right)} \left[ 2 \boldsymbol {f} _ {t} \left(\boldsymbol {x} _ {t}; \psi_ {t}\right) ^ {T} \left[ \boldsymbol {S} _ {t} \left(\boldsymbol {x} _ {t}\right) - \nabla_ {\boldsymbol {x} _ {t}} \log q _ {t} \left(\boldsymbol {x} _ {t} \mid \boldsymbol {x} _ {t + 1}; \phi_ {t}\right) \right] - \| \boldsymbol {f} _ {t} (\boldsymbol {x}; \psi_ {t}) \| _ {2} ^ {2} \right] \\ = - \mathbb {E} _ {q _ {t} \left(\boldsymbol {x} _ {t}, \boldsymbol {x} _ {t + 1}; \phi_ {\geq t}\right)} \| \boldsymbol {f} _ {t} (\boldsymbol {x}; \psi_ {t}) - \boldsymbol {S} _ {t} (\boldsymbol {x} _ {t}) + \nabla_ {\boldsymbol {x} _ {t}} \log q _ {t} \left(\boldsymbol {x} _ {t} \mid \boldsymbol {x} _ {t + 1}; \phi_ {t}\right) \| _ {2} ^ {2} + C \\ = - \mathbb {E} _ {q _ {t} (\pmb {x} _ {t}, \pmb {x} _ {t + 1}; \phi_ {\ge t})} \| \pmb {g} _ {t} (\pmb {x}; \psi_ {t}) - \nabla_ {\pmb {x} _ {t}} \log q _ {t} (\pmb {x} _ {t} | \pmb {x} _ {t + 1}; \phi_ {t}) \| _ {2} ^ {2} + C, \\ \end{array}
$$
where  $C$  is a term that does not contain  $\psi_t$ . Therefore, the minimax optimization problem is equivalent to
$$
\min  _ {\psi_ {t}} \quad \mathbb {E} _ {q _ {t} (\boldsymbol {x} _ {t}, \boldsymbol {x} _ {t + 1}; \phi_ {\geq t})} \| \boldsymbol {g} _ {t} (\boldsymbol {x} _ {t}; \psi_ {t}) - \nabla_ {\boldsymbol {x} _ {t}} \log q _ {t} (\boldsymbol {x} _ {t} | \boldsymbol {x} _ {t + 1}; \phi_ {t}) \| _ {2} ^ {2}.
$$
C.3 Joint training of HSIVI
As mentioned in Section 4.2, when parameter sharing scheme is used in the conditional layers for application to diffusion model acceleration, sequential training from  $t = T - 1$  to  $t = 0$  is not feasible. Therefore, we consider the following training objective
$$
\mathcal {L} _ {\mathrm {H S I V I -} f} (\phi) = \sum_ {t = 0} ^ {T - 1} \beta (t) \mathcal {L} _ {\mathrm {S I V I -} f} \left(p _ {t} \left(\boldsymbol {x} _ {t}\right) \| q _ {t} \left(\boldsymbol {x} _ {t}; \phi\right)\right).
$$
An intuitive method is to randomly sample a batch of time steps  $\{t_k\}_{k=1}^K$  and for each  $t_k$  train  $\mathcal{L}_{\mathrm{SIVI - f}}(p_{t_k}(\boldsymbol{x}_{t_k})\| q_{t_k}(\boldsymbol{x}_{t_k};\phi))$  directly. However, sequentially sampling  $\boldsymbol{x}_{t_k}$  through  $q(\boldsymbol{x}_i|\boldsymbol{x}_{i+1};\phi)$  from  $i = T-1$  to  $i = t_k$  is still necessary in this case, making it memory-consuming to preserve the computation graphs of the entire sampling process.
In order to reduce the cost of accumulating computation graphs, for each  $t$ , we treat  $q_{t + 1}(\pmb{x}_{t + 1};\phi)$  as a fixed mixing layer denoted by  $\tilde{q}_{t + 1}(\pmb{x}_{t + 1})$  and only fit the conditional layer  $q_{t}(\pmb{x}_{t}|\pmb{x}_{t + 1};\phi)$ . More specifically, for HSIVI-SM, we consider the following optimization problem
$$
\min  _ {\phi} \sum_ {t = 0} ^ {T - 1} \beta (t) \mathbb {E} _ {\tilde {q} _ {t} \left(\boldsymbol {x} _ {t}, \boldsymbol {x} _ {t + 1}; \phi\right)} \left[ \boldsymbol {S} _ {t} \left(\boldsymbol {x} _ {t}\right) - \boldsymbol {g} _ {t} \left(\boldsymbol {x} _ {t}; \psi\right) \right] ^ {T} \left[ \boldsymbol {S} _ {t} \left(\boldsymbol {x} _ {t}\right) + \boldsymbol {g} _ {t} \left(\boldsymbol {x} _ {t}; \psi\right) - 2 \nabla_ {\boldsymbol {x} _ {t}} \log q _ {t} \left(\boldsymbol {x} _ {t} \mid \boldsymbol {x} _ {t + 1}; \phi\right) \right], \tag {18}
$$
$$
\min  _ {\psi} \sum_ {t = 0} ^ {T - 1} \beta (t) \mathbb {E} _ {\bar {q} _ {t} \left(\boldsymbol {x} _ {t}, \boldsymbol {x} _ {t + 1}; \phi\right)} \| \boldsymbol {g} _ {t} (\boldsymbol {x} _ {t}; \psi) - \nabla_ {\boldsymbol {x} _ {t}} \log q _ {t} \left(\boldsymbol {x} _ {t} \mid \boldsymbol {x} _ {t + 1}; \phi\right) \| _ {2} ^ {2}, \tag {19}
$$
where  $\tilde{q}_t(\pmb{x}_t, \pmb{x}_{t+1}; \phi) = q_t(\pmb{x}_t | \pmb{x}_{t+1}; \phi) \tilde{q}_{t+1}(\pmb{x}_{t+1})$ . In what follows, we demonstrate that the above problem also ensures an accurate approximation of the target score function.
For equation (19), by the denoising score matching trick (Hyvärinen, 2005), the optimal point of  $\psi$ , denoted by  $\psi^{*}(\phi)$ , satisfies
$$
\boldsymbol {g} _ {t} (\boldsymbol {x} _ {t}; \psi^ {*} (\phi)) = \nabla_ {\boldsymbol {x} _ {t}} \log \tilde {q} _ {t} (\boldsymbol {x} _ {t}; \phi),
$$
where  $\tilde{q}_t(\pmb{x}_t;\phi) = \int q(\pmb{x}_t|\pmb{x}_{t + 1};\phi)\tilde{q} (\pmb{x}_{t + 1})\mathrm{d}\pmb{x}_{t + 1}$ . By plugging in the optimal point  $\psi^{*}(\phi)$ , each term in equation (18) is equivalent to
$$
\begin{array}{l} \mathbb {E} _ {\tilde {q} _ {t} \left(\boldsymbol {x} _ {t}, \boldsymbol {x} _ {t + 1}; \phi\right)} \left[ \boldsymbol {S} _ {t} \left(\boldsymbol {x} _ {t}\right) - \boldsymbol {g} _ {t} \left(\boldsymbol {x} _ {t}; \psi^ {*} (\phi)\right) \right] ^ {T} \left[ \boldsymbol {S} _ {t} \left(\boldsymbol {x} _ {t}\right) + \boldsymbol {g} _ {t} \left(\boldsymbol {x} _ {t}; \psi^ {*} (\phi)\right) - 2 \nabla_ {\boldsymbol {x} _ {t}} \log q _ {t} \left(\boldsymbol {x} _ {t} \mid \boldsymbol {x} _ {t + 1}; \phi\right) \right] \\ = \mathbb {E} _ {\tilde {q} _ {t} (\boldsymbol {x} _ {t}; \phi)} \left[ \boldsymbol {S} _ {t} ^ {2} (\boldsymbol {x} _ {t}) - \boldsymbol {g} _ {t} ^ {2} (\boldsymbol {x} _ {t}; \psi^ {*} (\phi)) \right] - 2 \iint \tilde {q} (\boldsymbol {x} _ {t + 1}) \left[ \boldsymbol {S} _ {t} (\boldsymbol {x} _ {t}) - \boldsymbol {g} _ {t} (\boldsymbol {x} _ {t}; \psi^ {*} (\phi)) \right] ^ {T} \nabla_ {\boldsymbol {x} _ {t}} q _ {t} (\boldsymbol {x} _ {t} | \boldsymbol {x} _ {t + 1}; \phi) d \boldsymbol {x} _ {t + 1} d \boldsymbol {x} _ {t} \\ = \mathbb {E} _ {\tilde {q} _ {t} (\boldsymbol {x} _ {t}; \phi)} \left[ \boldsymbol {S} _ {t} ^ {2} (\boldsymbol {x} _ {t}) - \boldsymbol {g} _ {t} ^ {2} (\boldsymbol {x} _ {t}; \psi^ {*} (\phi)) \right] - 2 \int \left[ \boldsymbol {S} _ {t} (\boldsymbol {x} _ {t}) - \boldsymbol {g} _ {t} (\boldsymbol {x} _ {t}; \psi^ {*} (\phi)) \right] ^ {T} \nabla_ {\boldsymbol {x} _ {t}} \tilde {q} _ {t} (\boldsymbol {x} _ {t}; \phi) d \boldsymbol {x} _ {t} \\ = \mathbb {E} _ {\tilde {q} _ {t} (\boldsymbol {x} _ {t}; \phi)} \left[ \boldsymbol {S} _ {t} ^ {2} (\boldsymbol {x} _ {t}) - 2 \boldsymbol {S} _ {t} (\boldsymbol {x} _ {t}) ^ {T} \nabla_ {\boldsymbol {x} _ {t}} \log \tilde {q} _ {t} (\boldsymbol {x} _ {t}; \phi) + \left(\nabla_ {\boldsymbol {x} _ {t}} \log \tilde {q} _ {t} (\boldsymbol {x} _ {t}; \phi)\right) ^ {2} \right] \\ = \mathbb {E} _ {\tilde {q} _ {t} (\boldsymbol {x} _ {t}; \phi)} \| \boldsymbol {S} _ {t} (\boldsymbol {x} _ {t}) - \nabla_ {\boldsymbol {x} _ {t}} \log \tilde {q} _ {t} (\boldsymbol {x} _ {t}; \phi) \| ^ {2}. \\ \end{array}
$$
Therefore, the global optimal point  $\phi^{*}$  also ensures that the score of the variational distribution fits the target score function.
Based on the training objectives (18) (19) mentioned above, we propose Algorithm 2 for joint training, which does not need to store the computation graphs of the sample sequences. Moreover, by assuming an increasing weighting function  $\beta(t)$ , we assign larger weights  $\beta(t)$  for those  $t$  close to  $T - 1$ , which tends to train the conditional layers that are close to  $T - 1$  first during the training, resembling the sequential training.
Algorithm 2 Hierarchical semi-implicit variational inference (joint training)
Input: Auxiliary bridge  $\{p_t(\pmb {x})\}_{t = 0}^{T - 1}$  ; a weighting function  $\beta (t)$  ; initial value of parameters  $\phi^{(0)}$    
Output: The optimal parameters  $\phi^*$    
Initialization:  $\phi \gets \phi^{(0)}$    
while not converge do Uniformly sample  $K$  time steps  $\{t_k\}_{k = 0}^K$  with replacement from  $\{0,\dots ,T - 1\}$  Sample a minibatch  $\{\pmb{x}_T^{(k)}\}_{k = 1}^K$  from the base distribution  $q_{T}(x)$  for  $k = 1,\ldots ,K$  and  $t_k <   T - 1$  do Sequentially sample  $\pmb{x}_{t_k + 1}^{(k)}$  through  $q(\pmb {x}_i|\pmb {x}_{i + 1};\phi_i)$  from  $i = T - 1$  to  $i = t_k + 1$  Detach the computation graphs from  $\{x_{t_k + 1}^{(k)}\}_{k = 1}^K$  end for Update  $\phi$  by optimizing the objective  $\sum_{k = 1}^{K}\beta (t_k)\mathcal{L}_{\mathrm{SIVI - f}}(p_{t_k}(\pmb{x}_{t_k})||q_{t_k}(\pmb{x}_{t_k};\phi))$  , where the  $k$  -th term is computed based on a single sample  $\pmb{x}_{t_k + 1}^{(k)}$    
end while  $\phi^{*}\gets \phi$
C.4  $\epsilon$ -training of HSIVI-SM
Another popular formulation of diffusion models is modeling the conditional noise  $\epsilon_{\theta}(\pmb{u}_s, s)$  by optimizing the DDPM loss in equation (14) where  $\pmb{u}_s = \sqrt{\alpha(s)} \pmb{u}_0 + \sqrt{1 - \alpha(s)} \pmb{\epsilon}$ , introduced as "ε-prediction" in Appendix A. Now, let us assume the diffusion bridge is constructed with VP-SDE and we have a pre-trained model of conditional noise  $\epsilon^*(\pmb{u}, s)$ . Similarly, we construct a sequence of noise models  $\{\pmb{\epsilon}_t^*(\pmb{x}_t)\}_{t=0}^{T-1}$  by letting  $\pmb{x}_t = \pmb{u}_{s_t}$  and  $\epsilon_t^*(\pmb{x}) = \epsilon_t^*(\pmb{x}, s_t)$  which forms a (generalized)  $T$ -layer diffusion bridge. We only discuss how  $\epsilon$ -training can be applied to joint training and the derivation for sequential training is similar. In what follows, we consider the transformation of the joint training objective  $\mathcal{L}_{\mathrm{HSIVI-SM}}$  for diffusion model acceleration.
By letting the weighting function  $\beta(t) = 1 - \alpha(s_t)$  and considering the reparametrization form (17) where  $\phi_t$  is replaced by  $\phi$ , the objective of HSIVI-SM takes the form
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {H S I V I - S M}} (\phi , \psi) = \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {\tilde {q} _ {t} \left(\boldsymbol {x} _ {t}, \boldsymbol {x} _ {t + 1}; \phi\right)} \left[ 2 \sqrt {\beta (t)} \boldsymbol {f} _ {t} \left(\boldsymbol {x} _ {t}; \psi\right) ^ {T} \left[ \sqrt {\beta (t)} \boldsymbol {S} _ {t} ^ {*} \left(\boldsymbol {x} _ {t}\right) + \sqrt {\beta (t)} \boldsymbol {\Sigma} _ {t} ^ {- 1 / 2} \left(\boldsymbol {x} _ {t + 1}; \phi\right) \boldsymbol {\epsilon}\right) \right] \\ \left. - \left\| \sqrt {\beta (t)} \boldsymbol {f} _ {t} (\boldsymbol {x} _ {t}; \psi) \right\| _ {2} ^ {2} \right]. \tag {20} \\ \end{array}
$$
where  $S_{t}^{*}(\boldsymbol{x}_{t})$  is a pre-trained score model. Note that we have  $\sqrt{\beta(t)} S_t^* (\boldsymbol{x}_t) = -\boldsymbol{\epsilon}_t^* (\boldsymbol{x}_t)$  by equation (15). Define
$$
\tilde {\boldsymbol {f}} _ {t} (\boldsymbol {x} _ {t}; \psi) = \sqrt {\beta (t)} \boldsymbol {f} _ {t} (\boldsymbol {x} _ {t}; \psi),
$$
$$
\tilde {\boldsymbol {\Sigma}} _ {t} \left(\boldsymbol {x} _ {t + 1}; \phi\right) = \boldsymbol {\Sigma} _ {t} \left(\boldsymbol {x} _ {t + 1}; \phi\right) / \beta (t).
$$
The HSIVI-SM objective (20) then takes the form
$$
\tilde {\mathcal {L}} _ {\mathrm {H S I V I - S M}} (\phi , \psi) = \sum_ {t = 0} ^ {T - 1} \mathbb {E} _ {\tilde {q} _ {t} \left(\boldsymbol {x} _ {t}, \boldsymbol {x} _ {t + 1}; \phi\right)} \left[ \right. 2 \tilde {\boldsymbol {f}} _ {t} \left(\boldsymbol {x} _ {t}; \psi\right) ^ {T} \left[ \right. - \boldsymbol {\epsilon} _ {t} ^ {*} \left(\boldsymbol {x} _ {t}\right) + \tilde {\boldsymbol {\Sigma}} _ {t} ^ {- 1 / 2} \left(\boldsymbol {x} _ {t + 1}; \phi\right) \boldsymbol {\epsilon}\left. \right)\left. \right] - \| \tilde {\boldsymbol {f}} _ {t} \left(\boldsymbol {x} _ {t}; \psi\right) \| _ {2} ^ {2} \left. \right] \tag {21}
$$
and we call it the objective for  $\epsilon$ -training. In our implementation of  $\epsilon$ -training, we directly parametrize  $\tilde{\pmb{f}}_t(\pmb{x}_t;\psi)$  and  $\tilde{\Sigma}_t(\pmb{x}_{t + 1};\phi)$  instead of  $\pmb {f}_t(\pmb {x}_t;\psi)$  and  $\pmb {\Sigma}_t(\pmb {x}_{t + 1};\phi)$ . The objective (21) is more numerically stable since the magnitude of  $\tilde{\Sigma}_t(\pmb{x}_{t + 1};\phi)$  is generally larger than  $\pmb {\Sigma}_t(\pmb{x}_{t + 1};\phi)$ .
C.5 Complexity comparison of HSIVI
For methods that we discussed in SIVI variants, which use a single conditional layer (i.e.,  $T = 1$ ) and hence would be much cheaper to sample from than HSIVI-SM that uses multiple layers  $T > 1$ . For methods that we discussed for diffusion models, the computational complexity would be similar if they had the same  $T$ . That is because we used the same neural network architecture for the conditional layers in HSIVI and the score nets in diffusion models. We have a comparison of the sampling time of different methods in Figure 13.
Figure 8: Upper row: Sample trajectories progressively generated by 5-layer HSIVI-LB guided by diffusion bridge. Bottom row: Sample trajectories progressively generated by 5-layer HSIVI-SM guided by diffusion bridge.
Figure 9: The posterior estimates for conditioned diffusion obtained by SIVI-LB and 5-layer HSIVI-LB. For each method, we collect 100,000 samples to calculate the sample mean and confidence interval.
D Additional results of experiments
D.1 Gaussian mixture model
For HSIVI on the Gaussian mixture model, the auxiliary distributions can also be constructed with diffusion bridge in Example 2. Concretely, the diffusion bridge is constructed by
$$
\boldsymbol {x} _ {t} \mid \boldsymbol {x} _ {0} \sim \mathcal {N} (\sqrt {\alpha_ {t}} \boldsymbol {x} _ {0}, (1 - \alpha_ {t}) \mathbf {I}), \quad \boldsymbol {x} _ {0} \sim p _ {0} (\boldsymbol {x} _ {0}).
$$