SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 1 Segment-wise forwarding-backwarding algorithm (PyTorch-like pseudo-codes)
1: Input:  $f_{r}, f_{d}, \partial \mathbf{L} / \partial \mathbf{x}_{0}', \mathbf{x}_{i}, \mathbf{x}_{i}' (i \in [T])$   
2: Output:  $\partial \mathcal{L} / \partial \mathbf{x}_0$   
3: for  $t = 1$  to  $T$  do  
4: Creat_Graph( $f_{r}(\mathbf{x}_{t}') \to \mathbf{x}_{t-1}')$   
5:  $\mathcal{L}' \gets (\partial \mathcal{L} / \partial \mathbf{x}_{t-1}') (f_{r}(\mathbf{x}_{t}))$   
6:  $\partial \mathcal{L} / \partial \mathbf{x}_{t}' \gets \text{auto\_grad}(\mathcal{L}', \mathbf{x}_{t}')$   
7: Release_Graph( $f_{r}(\mathbf{x}_{t}') \to \mathbf{x}_{t-1}')$   
8: end for  
9:  $\partial \mathcal{L} / \partial \mathbf{x}_{T} \gets \partial \mathcal{L} / \partial \mathbf{x}_{T}$   
10: for  $t = T - 1$  to 0 do  
11: Creat_Graph( $f_{d}(\mathbf{x}_{t}) \to \mathbf{x}_{t+1}$ )  
12:  $\mathcal{L}' \gets (\partial \mathcal{L} / \partial \mathbf{x}_{t+1})(f_{d}(\mathbf{x}_{t}))$   
13:  $\partial \mathcal{L} / \partial \mathbf{x}_{t} \gets \text{auto\_grad}(\mathcal{L}', \mathbf{x}_{t})$   
14: Release_Graph( $f_{d}(\mathbf{x}_{t}) \to \mathbf{x}_{t+1}$ )  
15: end for
B Efficient Gradient Backpropagation
In this section, we provide the PyTorch-like pseudo-codes of the segment-wise forwarding-backwarding algorithm. At each time step  $t$  in the reverse process, we only need to store the gradient  $\partial \mathcal{L} / \partial \mathbf{x}_t'$ , the intermediate sample  $\mathbf{x}_{t + 1}'$  and the model  $f_{r}$  to construct the computational graph. When we backpropagate the gradients at the next time step  $t + 1$ , the computational graph at
time step  $t$  will no longer be reused, and thus, we can release the memory of the graph at time step  $t$ . Therefore, we only have one segment of the computational graph used for gradient backpropagation in the memory at each time step. We can similarly backpropagate the gradients in the diffusion process.
C.1 Proof of Thm. 1
Assumption C.1. Consider adversarial sample  $\tilde{\mathbf{x}}_0\coloneqq \mathbf{x}_0 + \delta$  , where  $\mathbf{x}_0$  is the clean example and  $\delta$  is the perturbation.  $p_t(\mathbf{x}),p_t'(\mathbf{x}),q_t(\mathbf{x}),q_t'(\mathbf{x})$  are the distribution of  $\mathbf{x}_t,\mathbf{x}_t',\tilde{\mathbf{x}}_t,\tilde{\mathbf{x}}_t'$  where  $\mathbf{x}_t^\prime$  represents the reconstruction of  $\mathbf{x}_t$  at time step  $t$  in the reverse process. We consider a score-based diffusion model with a well-trained score-based model  $s_\theta$  parameterized by  $\theta$  with the clean training distribution. Therefore, we assume that  $s_\theta$  can achieve a low score-matching loss given a clean sample and reconstruct it in the reverse process:
$$
\left\| \nabla_ {\mathbf {x}} \log p _ {t} (\mathbf {x}) - s _ {\theta} (\mathbf {x}, t) \right\| _ {2} ^ {2} \leq L _ {u} \tag {12}
$$
$$
D _ {T V} \left(p _ {t}, p _ {t} ^ {\prime}\right) \leq \epsilon_ {r e} \tag {13}
$$
where  $D_{TV}(\cdot, \cdot)$  is the total variation distance.  $L_u$  and  $\epsilon_{re}$  are two small constants that characterize the score-matching loss and the reconstruction error.
Assumption C.2. We assume the score function of data distribution is bounded by  $M$ :
$$
\left\| \nabla_ {\mathbf {x}} \log p _ {t} (\mathbf {x}) \right\| _ {2} \leq M, \left\| \nabla_ {\mathbf {x}} \log q _ {t} (\mathbf {x}) \right\| _ {2} \leq M \tag {14}
$$
Lemma C.1. Consider adversarial sample  $\tilde{\mathbf{x}}_0\coloneqq \mathbf{x}_0 + \delta$  , where  $\mathbf{x}_0$  is the clean example and  $\delta$  is the perturbation.  $p_t(\mathbf{x}),p_t'(\mathbf{x}),q_t(\mathbf{x}),q_t'(\mathbf{x})$  are the distribution of  $\mathbf{x}_t,\mathbf{x}_t',\tilde{\mathbf{x}}_t,\tilde{\mathbf{x}}_t'$  where  $\mathbf{x}_t^\prime$  represents the reconstruction of  $\mathbf{x}_t$  in the reverse process. Given a VP-SDE parameterized by  $\beta (\cdot)$  and the score-based model  $s_\theta$  with Assumption C.2, we have:
$$
D _ {K L} \left(p _ {t} ^ {\prime}, q _ {t} ^ {\prime}\right) = \frac {1}{2} \int_ {t} ^ {T} \beta (s) \mathbb {E} _ {\mathbf {x} \mid \mathbf {x} _ {0}} \| \nabla_ {\mathbf {x}} \log p _ {s} ^ {\prime} (\mathbf {x}) - \nabla_ {\mathbf {x}} \log q _ {s} ^ {\prime} (\mathbf {x}) \| _ {2} ^ {2} d s + 4 M ^ {2} \int_ {t} ^ {T} \beta (s) d s \tag {15}
$$
Proof. The reverse process of VP-SDE can be formulated as follows:
$$
d \mathbf {x} = f _ {r e v} (\mathbf {x}, t, p _ {t}) d t + g _ {r e v} (t) d \mathbf {w}, \text {w h e r e} f _ {r e v} (\mathbf {x}, t, p _ {t}) = - \frac {1}{2} \beta (t) \mathbf {x} - \beta (t) \nabla_ {\mathbf {x}} \log p _ {t} (\mathbf {x}), g _ {r e v} (t) = \sqrt {\beta (t)} \tag {16}
$$
Using the Fokker-Planck equation [44] in Equation (16), we have:
$$
\begin{array}{l} \frac {\partial p _ {t} ^ {\prime} (\mathbf {x})}{\partial t} = - \nabla_ {\mathbf {x}} \left(f _ {r e v} (\mathbf {x}, t, p _ {t}) p _ {t} ^ {\prime} (\mathbf {x}) - \frac {1}{2} g _ {r e v} ^ {2} (t) \nabla_ {\mathbf {x}} p _ {t} ^ {\prime} (x)\right) (17) \\ = \nabla_ {\mathbf {x}} \left(\left(\frac {1}{2} g _ {r e v} ^ {2} (t) \nabla_ {\mathbf {x}} \log p _ {t} ^ {\prime} (\mathbf {x}) - f _ {r e v} (\mathbf {x}, t, p _ {t})\right) p _ {t} ^ {\prime} (\mathbf {x})\right) (18) \\ \end{array}
$$
Similarly, applying the Fokker-Planck equation on the reverse SDE for  $q_{t}^{\prime}(\mathbf{x})$ , we can get:
$$
\frac {\partial q _ {t} ^ {\prime} (\mathbf {x})}{\partial t} = \nabla_ {\mathbf {x}} \left(\left(\frac {1}{2} g _ {r e v} ^ {2} (t) \nabla_ {\mathbf {x}} \log q _ {t} ^ {\prime} (\mathbf {x}) - f _ {r e v} (\mathbf {x}, t, q _ {t})\right) q _ {t} ^ {\prime} (\mathbf {x})\right) \tag {19}
$$
We use the notation  $h_p(\mathbf{x}) = \frac{1}{2} g_{rev}^2(t) \nabla_{\mathbf{x}} \log p_t'(\mathbf{x}) - f_{rev}(\mathbf{x}, t, p_t)$  and  $h_q(x) = \frac{1}{2} g_{rev}^2(t) \nabla_{\mathbf{x}} \log q_t'(\mathbf{x}) - f_{rev}(\mathbf{x}, t, q_t)$ . Then according to [34] (Theorem A.1), under the assumption that  $p_t'(\mathbf{x})$  and  $q_t'(\mathbf{x})$  are smooth and fast decaying (i.e.,  $\lim_{\mathbf{x}_i \to \infty} [p_t'(\mathbf{x}) \partial \log p'(\mathbf{x}) / \partial \mathbf{x}_i] = 0$ ),  $\lim_{\mathbf{x}_i \to \infty} [q_t'(\mathbf{x}) \partial \log q'(\mathbf{x}) / \partial \mathbf{x}_i] = 0$ ), we have:
$$
\frac {\partial D _ {K L} \left(p _ {t} ^ {\prime} , q _ {t} ^ {\prime}\right)}{\partial t} = - \int p _ {t} ^ {\prime} (x) \left[ h _ {p} (\mathbf {x}, t) - h _ {q} (\mathbf {x}, t) \right] ^ {T} \left[ \nabla_ {\mathbf {x}} \log p _ {t} ^ {\prime} (\mathbf {x}) - \nabla_ {\mathbf {x}} \log q _ {t} ^ {\prime} (\mathbf {x}) \right] d \mathbf {x} \tag {20}
$$
Plugging in Equations (18) and (19), we have:
$$
\begin{array}{l} \frac {\partial D _ {K L} (p _ {t} ^ {\prime} , q _ {t} ^ {\prime})}{\partial t} = - \int p _ {t} ^ {\prime} (\mathbf {x}) \left(\frac {1}{2} g _ {r e v} ^ {2} (t) \| \nabla_ {\mathbf {x}} \log p _ {t} ^ {\prime} (\mathbf {x}) - \nabla_ {\mathbf {x}} \log q _ {t} ^ {\prime} (\mathbf {x}) \| _ {2} ^ {2} \right. \\ + \beta (t) \left[ \nabla_ {\mathbf {x}} \log p _ {t} (\mathbf {x}) - \nabla_ {\mathbf {x}} \log q _ {t} (\mathbf {x}) \right] ^ {T} \left[ \nabla_ {\mathbf {x}} \log p _ {t} ^ {\prime} (\mathbf {x}) - \nabla_ {\mathbf {x}} \log q _ {t} ^ {\prime} (\mathbf {x}) \right]) d \mathbf {x} \tag {21} \\ \end{array}
$$
Finally, we can derive as follows:
$$
\begin{array}{l} D _ {K L} \left(p _ {t} ^ {\prime}, q _ {t} ^ {\prime}\right) = \int_ {t} ^ {T} \int_ {\mathcal {X}} \left(p _ {s} ^ {\prime} (\mathbf {x}) \left(\frac {1}{2} g _ {r e v} ^ {2} (s) \| \nabla_ {\mathbf {x}} \log p _ {s} ^ {\prime} (\mathbf {x}) - \nabla_ {\mathbf {x}} \log q _ {s} ^ {\prime} (\mathbf {x}) \| _ {2} ^ {2} \right. \right. (22) \\ + \beta (s) \left[ \nabla_ {\mathbf {x}} \log p _ {s} (\mathbf {x}) - \nabla_ {\mathbf {x}} \log q _ {s} (\mathbf {x}) \right] ^ {T} \left[ \nabla_ {\mathbf {x}} \log p _ {s} ^ {\prime} (\mathbf {x}) - \nabla_ {\mathbf {x}} \log q _ {s} ^ {\prime} (\mathbf {x}) \right]) d \mathbf {x} d s (23) \\ \leq \int_ {t} ^ {T} \left(\frac {1}{2} g _ {r e v} ^ {2} (s) \mathbb {E} _ {\mathbf {x} | \mathbf {x} _ {0}} \| \nabla_ {\mathbf {x}} \log p _ {s} ^ {\prime} (\mathbf {x}) - \nabla_ {\mathbf {x}} \log q _ {s} ^ {\prime} (\mathbf {x}) \| _ {2} ^ {2} + 4 \beta (s) M ^ {2}\right) d s (24) \\ = \frac {1}{2} \int_ {t} ^ {T} \beta (s) \mathbb {E} _ {\mathbf {x} \mid \mathbf {x} _ {0}} \| \nabla_ {\mathbf {x}} \log p _ {s} ^ {\prime} (\mathbf {x}) - \nabla_ {\mathbf {x}} \log q _ {s} ^ {\prime} (\mathbf {x}) \| _ {2} ^ {2} d s + 4 M ^ {2} \int_ {t} ^ {T} \beta (s) d s (25) \\ \end{array}
$$
Theorem 2 (Thm. 1 in the main text). Consider adversarial sample  $\tilde{\mathbf{x}}_0 \coloneqq \mathbf{x}_0 + \delta$ , where  $\mathbf{x}_0$  is the clean example and  $\delta$  is the perturbation.  $p_t(\mathbf{x}), p_t'(\mathbf{x}), q_t(\mathbf{x}), q_t'(\mathbf{x})$  are the distribution of  $\mathbf{x}_t, \mathbf{x}_t', \tilde{\mathbf{x}}_t, \tilde{\mathbf{x}}_t'$  where  $\mathbf{x}_t'$  represents the reconstruction of  $\mathbf{x}_t$  in the reverse process.  $D_{TV}(\cdot, \cdot)$  measures the total variation distance. Given a VP-SDE parameterized by  $\beta(\cdot)$  and the score-based model  $s_\theta$  with mild assumptions that  $\| \nabla_{\mathbf{x}} \log p_t(\mathbf{x}) - s_\theta(\mathbf{x}, t) \|_2^2 \leq L_u$ ,  $D_{TV}(p_t, p_t') \leq \epsilon_{re}$ , and a bounded score function by  $M$  (specified with details in Appendix C.1), we have:
$$
\begin{array}{l} D _ {T V} \left(q _ {t}, q _ {t} ^ {\prime}\right) \leq \frac {1}{2} \sqrt {\mathbb {E} _ {t , \mathbf {x} | \mathbf {x} _ {0}} \left\| \boldsymbol {s} _ {\theta} (\mathbf {x} , t) - \nabla_ {\mathbf {x}} \log q _ {t} ^ {\prime} (\mathbf {x}) \right\| _ {2} ^ {2} + C _ {1}} \tag {26} \\ + \sqrt {2 - 2 \exp \{- C _ {2} \| \delta \| _ {2} ^ {2} \}} + \epsilon_ {r e} \\ \end{array}
$$
$$
w h e r e C _ {1} = \left(L _ {u} + 8 M ^ {2}\right) \int_ {t} ^ {T} \beta (t) d t, C _ {2} = \frac {1}{8 \left(1 - \Pi_ {s = 1} ^ {t} \left(1 - \beta_ {s}\right)\right)}.
$$
Proof. Since we consider VP-SDE here, we have:
$$
f (\mathbf {x}, t) = - \frac {1}{2} \beta (t) \mathbf {x}, \quad g (t) = \sqrt {\beta (t)} \tag {27}
$$
$$
f _ {r e v} (\mathbf {x}, t) = - \frac {1}{2} \beta (t) \mathbf {x} - \beta (t) \nabla_ {\mathbf {x}} \log p _ {t} (\mathbf {x}), \quad g _ {r e v} (t) = \sqrt {\beta (t)} \tag {28}
$$
Using the triangular inequality, the total variation distance between  $q_{t}$  and  $q_{t}^{\prime}$  can be decomposed as:
$$
D _ {T V} \left(q _ {t}, q _ {t} ^ {\prime}\right) \leq D _ {T V} \left(q _ {t}, p _ {t}\right) + D _ {T V} \left(p _ {t}, p _ {t} ^ {\prime}\right) + D _ {T V} \left(q _ {t} ^ {\prime}, p _ {t} ^ {\prime}\right) \tag {29}
$$
According to Assumption C.1, we have  $D_{TV}(p_t,p_t^{\prime})\leq \epsilon_{re}$  and thus, we only need to upper bound  $D_{TV}(q_t,p_t)$  and  $D_{TV}(q_t^{\prime},p_t^{\prime})$ , respectively.
Using the notation  $\alpha_{t} \coloneqq 1 - \beta(t)$  and  $\bar{\alpha}_{t} \coloneqq \Pi_{s=1}^{t} \alpha_{s}$ , we have:
$$
\mathbf {x} _ {t} \sim p _ {t} := \mathcal {N} \left(\mathbf {x} _ {t}; \sqrt {\bar {\alpha} _ {t}} \mathbf {x} _ {0}, (1 - \bar {\alpha} _ {t}) \mathbf {I}\right), \quad \tilde {\mathbf {x}} _ {t} \sim q _ {t} := \mathcal {N} \left(\tilde {\mathbf {x}} _ {t}; \sqrt {\bar {\alpha} _ {t}} \tilde {\mathbf {x}} _ {0}, (1 - \bar {\alpha} _ {t}) \mathbf {I}\right) \tag {30}
$$
Therefore, we can upper bound the total variation distance between  $q_{t}$  and  $p_t$  as follows:
$$
\begin{array}{l} D _ {T V} \left(q _ {t}, p _ {t}\right) \leq \sqrt [ (a) ]{2} H \left(\mathbf {x} _ {t}, \tilde {\mathbf {x}} _ {t}\right) (31) \\ \stackrel {(b)} {=} \sqrt {2} \sqrt {1 - \exp \left\{- \frac {1}{8 \left(1 - \bar {\alpha} _ {t}\right)} \delta^ {T} \delta \right\}} (32) \\ = \sqrt {2 - 2 \exp \left\{- \frac {1}{8 \left(1 - \bar {\alpha} _ {t}\right)} \| \delta \| _ {2} ^ {2} \right\}} (33) \\ \end{array}
$$
where we leverage the inequality between the Hellinger distance  $H(\cdot, \cdot)$  and total variation distance in Equation (31)(a) and we plug in the closed form of Hellinger distance between two Gaussian distribution [14] parameterized by  $\mu_1, \Sigma_1, \mu_2, \Sigma_2$  in Equation (32)(b):
$$
H \left(\mathcal {N} \left(\mu_ {1}, \Sigma_ {1}\right), \mathcal {N} \left(\mu_ {2}, \Sigma_ {2}\right)\right) ^ {2} = 1 - \frac {\operatorname* {d e t} \left(\Sigma_ {1}\right) ^ {1 / 4} \operatorname* {d e t} \left(\Sigma_ {2}\right) ^ {1 / 4}}{\operatorname* {d e t} \left(\frac {\Sigma_ {1} + \Sigma_ {2}}{2}\right) ^ {1 / 2}} \exp \left\{- \frac {1}{8} \left(\mu_ {1} - \mu_ {2}\right) ^ {T} \left(\frac {\Sigma_ {1} + \Sigma_ {2}}{2}\right) ^ {- 1} \left(\mu_ {1} - \mu_ {2}\right) \right\} \tag {34}
$$
Then, we will upper bound  $D_{TV}(p_t', q_t')$ . We first leverage Pinker's inequality to upper bound the total variation distance with the KL-divergence:
$$
D _ {T V} \left(p _ {t} ^ {\prime}, q _ {t} ^ {\prime}\right) \leq \sqrt {\frac {1}{2} D _ {K L} \left(p _ {t} ^ {\prime} , q _ {t} ^ {\prime}\right)} \tag {35}
$$
Then we plug in the results in Lemma C.1 to upper bound  $KL(p_t', q_t')$  and it follows that:
$$
\begin{array}{l} D _ {T V} \left(p _ {t} ^ {\prime}, q _ {t} ^ {\prime}\right) (36) \\ \leq \sqrt {\frac {1}{2} D _ {K L} \left(p _ {t} ^ {\prime} , q _ {t} ^ {\prime}\right)} (37) \\ \leq \sqrt {\frac {1}{4} \int_ {t} ^ {T} \beta (s) \mathbb {E} _ {\mathbf {x} | \mathbf {x} _ {0}} \| \nabla_ {\mathbf {x}} \log p _ {s} ^ {\prime} (\mathbf {x}) - \nabla_ {\mathbf {x}} \log q _ {s} ^ {\prime} (\mathbf {x}) \| _ {2} ^ {2} d s + 2 M ^ {2} \int_ {t} ^ {T} \beta (s) d s} (38) \\ \leq \sqrt {\frac {1}{4} \int_ {t} ^ {T} \beta (s) \mathbb {E} _ {\mathbf {x} | \mathbf {x} _ {0}} [ \| \nabla_ {\mathbf {x}} \log p _ {s} ^ {\prime} (\mathbf {x}) - s _ {\theta} (\mathbf {x} , s) \| _ {2} ^ {2} + \| s _ {\theta} (\mathbf {x} , s) - \nabla_ {\mathbf {x}} \log q _ {s} ^ {\prime} (\mathbf {x}) \| _ {2} ^ {2} ] d s + 2 M ^ {2} \int_ {t} ^ {T} \beta (s) d s} (39) \\ \stackrel {(a)} {\leq} \sqrt {\left(\frac {L _ {u}}{4} + 2 M ^ {2}\right) \int_ {t} ^ {T} \beta (s) d s + \frac {1}{4} \mathbb {E} _ {t , \mathbf {x} | \mathbf {x} _ {0}} \| \boldsymbol {s} _ {\theta} (\mathbf {x} , t) - \nabla_ {\mathbf {x}} \log q _ {t} ^ {\prime} (\mathbf {x}) \| _ {2} ^ {2}} (40) \\ \end{array}
$$
where in Equation (40)(a), we leverage the fact that  $\beta (\cdot)$  is bounded in [0, 1].
Combining Equations (29), (33) and (40), we can finally get:
$$
D _ {T V} \left(q _ {t}, q _ {t} ^ {\prime}\right) \leq \sqrt {\frac {1}{4} \mathbb {E} _ {t , \mathbf {x} | \mathbf {x} _ {0}} \| s _ {\theta} (\mathbf {x} , t) - \nabla_ {\mathbf {x}} \log q _ {t} ^ {\prime} (\mathbf {x}) \| _ {2} ^ {2} + C _ {1}} + \sqrt {2 - 2 \exp \{- C _ {2} \| \delta \| _ {2} ^ {2} \}} + \epsilon_ {r e} \tag {41}
$$
where  $C_1 = \left(\frac{L_u}{4} + 2M^2\right)\int_t^T\beta(s)ds$  and  $C_2 = \frac{1}{8(1 - \Pi_{s=1}^t(1 - \beta_s))}$ .
C.2 Connection between the deviated-reconstruction loss and the density gradient loss for DDPM
Theorem 3. Consider adversarial sample  $\tilde{\mathbf{x}}_0\coloneqq \mathbf{x}_0 + \delta$  where  $\mathbf{x}_0$  is the clean example and  $\delta$  is the perturbation.  $p_t(\mathbf{x}),p_t'(\mathbf{x}),q_t(\mathbf{x}),q_t'(\mathbf{x})$  are the distribution of  $\mathbf{x}_t,\mathbf{x}_t',\tilde{\mathbf{x}}_t,\tilde{\mathbf{x}}_t'$  where  $\mathbf{x}_t^\prime$  represents the reconstruction of  $\mathbf{x}_t$  in the reverse process. Given a DDPM parameterized by  $\beta (\cdot)$  and the function approximator  $\boldsymbol {s}_{\theta}$  with the mild assumptions that  $\| s_{\theta}(\mathbf{x},t) - \epsilon (\mathbf{x}_t,t)\| _2^2\leq L_u$ $D_{TV}(p_t,p_t')\leq \epsilon_{re}$  and a bounded score function by  $M$  (i.e.,  $\| \epsilon (\mathbf{x},t)\| _2\leq M$ ) where  $\epsilon (\cdot ,\cdot)$  represents the mapping function of the true perturbation, we have:
$$
\begin{array}{l} D _ {T V} \left(q _ {t}, q _ {t} ^ {\prime}\right) \leq \sqrt {2 - 2 \exp \left\{- C _ {2} \left(\sum_ {k = t + 1} ^ {T} \lambda (k , t) \| \boldsymbol {s} _ {\theta} \left(\tilde {\mathbf {x}} _ {k} ^ {\prime} , k\right) - \epsilon \left(\tilde {\mathbf {x}} _ {k} ^ {\prime} , k\right) \| _ {2} + C _ {1} \| \delta \| _ {2} + (\sqrt {L _ {u}} + 2 M) \sum_ {k = t + 1} ^ {T} \lambda (k , t)\right) ^ {2} \right\}} \tag {42} \\ + \sqrt {2 - 2 \exp \{- \frac {1}{8} (1 - \Pi_ {s = 1} ^ {t} (1 - \beta_ {s})) \| \delta \| _ {2} ^ {2} \}} + \epsilon_ {r e} \\ \end{array}
$$
where  $C_1 = \left(\Pi_{s=t+1}^T \sqrt{\Pi_{k=1}^s (1 - \beta_k)}\right) \sqrt{\Pi_{s=1}^T (1 - \beta_s)}$ ,  $C_2 = \frac{1 - \Pi_{s=1}^t (1 - \beta_s)}{8(1 - \Pi_{s=1}^{t-1} (1 - \beta_s)) \beta_t}$ , and
$$
\lambda (k, t) = \frac {\beta_ {k} \Pi_ {i = t + 1} ^ {k - 1} \sqrt {\Pi_ {s = 1} ^ {i} (1 - \beta_ {s})}}{\sqrt {1 - \Pi_ {s = 1} ^ {k} (1 - \beta_ {s})}}.
$$
Proof. For ease of notation, we use the notation:  $\alpha_{t} := 1 - \beta_{t}$  and  $\bar{\alpha}_{t} := \Pi_{s=1}^{t} \alpha_{s}$ . From the DDPM sampling process [25], we know that:
$$
\mathbf {x} _ {t - 1} ^ {\prime} \sim p _ {t} ^ {\prime} := \frac {1}{\sqrt {\bar {\alpha} _ {t}}} \left(\mathbf {x} _ {t} ^ {\prime} - \frac {1 - \alpha_ {t}}{\sqrt {1 - \bar {\alpha} _ {t}}} s _ {\theta} \left(\mathbf {x} _ {t} ^ {\prime}, t\right)\right) + \sigma_ {t} \mathbf {z} \tag {43}
$$
$$
\tilde {\mathbf {x}} _ {t - 1} ^ {\prime} \sim q _ {t} ^ {\prime} := \frac {1}{\sqrt {\bar {\alpha} _ {t}}} \left(\tilde {\mathbf {x}} _ {t} ^ {\prime} - \frac {1 - \alpha_ {t}}{\sqrt {1 - \bar {\alpha} _ {t}}} s _ {\theta} \left(\tilde {\mathbf {x}} _ {t} ^ {\prime}, t\right)\right) + \sigma_ {t} \mathbf {z} \tag {44}
$$
where  $\sigma_t^2 = \frac{1 - \bar{\alpha}_{t-1}}{1 - \bar{\alpha}_t} \beta_t$ .
$\pmb{\mu}_{t,q}$  and  $\pmb{\mu}_{t,p}$  represent the mean of the distribution  $q_{t}^{\prime}$  and  $p_t^\prime$  , respectively. Then from Equations (43) and (44), we have:
$$
\boldsymbol {\mu} _ {t, q} - \boldsymbol {\mu} _ {t, p} = \frac {1}{\sqrt {\bar {\alpha} _ {t}}} \left(\boldsymbol {\mu} _ {t - 1, q} - \boldsymbol {\mu} _ {t - 1, p}\right) - \frac {1 - \alpha_ {t}}{\sqrt {\bar {\alpha} _ {t}} \sqrt {1 - \bar {\alpha} _ {t}}} \left(\boldsymbol {s} _ {\theta} \left(\tilde {\mathbf {x}} _ {t} ^ {\prime}, t\right) - \boldsymbol {s} _ {\theta} \left(\mathbf {x} _ {t} ^ {\prime}, t\right)\right) \tag {45}
$$
Applying Equation (45) iteratively, we get:
$$
\boldsymbol {\mu} _ {T, q} - \boldsymbol {\mu} _ {T, p} = \frac {1}{\Pi_ {s = t} ^ {T} \sqrt {\bar {\alpha} _ {s}}} \left(\boldsymbol {\mu} _ {t - 1, q} - \boldsymbol {\mu} _ {t - 1, p}\right) - \sum_ {k = t} ^ {T} \frac {1 - \alpha_ {k}}{\sqrt {1 - \bar {\alpha} _ {k}} \Pi_ {i = k} ^ {T} \sqrt {\bar {\alpha} _ {i}}} \left(\boldsymbol {s} _ {\theta} \left(\tilde {\mathbf {x}} _ {k} ^ {\prime}, k\right) - \boldsymbol {s} _ {\theta} \left(\mathbf {x} _ {k} ^ {\prime}, k\right)\right) \tag {46}
$$
On the other hand,  $\pmb{\mu}_{T,q} - \pmb{\mu}_{T,p}$  can be formulated explicitly considering the Gaussian distribution at time step  $T$  in the diffusion process:
$$
\boldsymbol {\mu} _ {T, q} - \boldsymbol {\mu} _ {T, p} = \sqrt {\bar {\alpha} _ {T}} \left(\tilde {\mathbf {x}} _ {0} - \mathbf {x} _ {0}\right) = \sqrt {\bar {\alpha} _ {T}} \delta \tag {47}
$$
Combining Equations (46) and (47), we can derive that:
$$
\begin{array}{l} \left\| \boldsymbol {\mu} _ {T, q} - \boldsymbol {\mu} _ {T, p} \right\| _ {2} (48) \\ = \left(\Pi_ {s = t + 1} ^ {T} \sqrt {\bar {\alpha} _ {s}}\right) \sqrt {\bar {\alpha} _ {T}} \| \delta \| _ {2} + \sum_ {k = t + 1} ^ {T} \frac {(1 - \alpha_ {k}) \Pi_ {i = t + 1} ^ {k - 1} \sqrt {\bar {\alpha} _ {i}}}{\sqrt {1 - \bar {\alpha} _ {k}}} \| \boldsymbol {s} _ {\theta} \left(\tilde {\mathbf {x}} _ {k} ^ {\prime}, k\right) - \boldsymbol {s} _ {\theta} \left(\mathbf {x} _ {k} ^ {\prime}, k\right) \| _ {2} (49) \\ \leq \left(\Pi_ {s = t + 1} ^ {T} \sqrt {\bar {\alpha} _ {s}}\right) \sqrt {\bar {\alpha} _ {T}} \| \delta \| _ {2} + \sum_ {k = t + 1} ^ {T} \frac {(1 - \alpha_ {k}) \Pi_ {i = t + 1} ^ {k - 1} \sqrt {\bar {\alpha} _ {i}}}{\sqrt {1 - \bar {\alpha} _ {k}}} \left(\| \boldsymbol {s} _ {\theta} (\tilde {\mathbf {x}} _ {k} ^ {\prime}, k) - \epsilon (\mathbf {x} _ {k} ^ {\prime}, k) \| _ {2} + \| \epsilon (\mathbf {x} _ {k} ^ {\prime}, k) - \boldsymbol {s} _ {\theta} (\mathbf {x} _ {k} ^ {\prime}, k)) \| _ {2}\right) (50) \\ \leq \left(\Pi_ {s = t + 1} ^ {T} \sqrt {\bar {\alpha} _ {s}}\right) \sqrt {\bar {\alpha} _ {T}} \| \delta \| _ {2} + \sqrt {L _ {u}} \sum_ {k = t + 1} ^ {T} \lambda (k, t) + \sum_ {k = t + 1} ^ {T} \lambda (k, t) \left(\| \boldsymbol {s} _ {\theta} \left(\tilde {\mathbf {x}} _ {k} ^ {\prime}, k\right) - \epsilon \left(\tilde {\mathbf {x}} _ {k} ^ {\prime}, k\right) \| _ {2} + \| \epsilon \left(\tilde {\mathbf {x}} _ {k} ^ {\prime}, k\right) - \epsilon \left(\mathbf {x} _ {k} ^ {\prime}, k\right) \| _ {2}\right) (51) \\ \leq \left(\Pi_ {s = t + 1} ^ {T} \sqrt {\bar {\alpha} _ {s}}\right) \sqrt {\bar {\alpha} _ {T}} \| \delta \| _ {2} + (\sqrt {L _ {u}} + 2 M) \sum_ {k = t + 1} ^ {T} \lambda (k, t) + \sum_ {k = t + 1} ^ {T} \lambda (k, t) \| \boldsymbol {s} _ {\theta} \left(\tilde {\mathbf {x}} _ {k} ^ {\prime}, k\right) - \epsilon \left(\tilde {\mathbf {x}} _ {k} ^ {\prime}, k\right) \| _ {2} (52) \\ \end{array}
$$
where  $\lambda (k,t) = \frac{(1 - \alpha_k)\Pi_{i = t + 1}^{k - 1}\sqrt{\bar{\alpha}_i}}{\sqrt{1 - \bar{\alpha}_k}}$
We then leverage the closed form formulation of the Hellinger distance between two Gaussian distributions [14] parameterized by  $\mu_1, \Sigma_1, \mu_2, \Sigma_2$ :
$$
H ^ {2} \left(\mathcal {N} \left(\mu_ {1}, \Sigma_ {1}\right), \mathcal {N} \left(\mu_ {2}, \Sigma_ {2}\right)\right) = 1 - \frac {\operatorname* {d e t} \left(\Sigma_ {1}\right) ^ {1 / 4} \operatorname* {d e t} \left(\Sigma_ {2}\right) ^ {1 / 4}}{\operatorname* {d e t} \left(\frac {\Sigma_ {1} + \Sigma_ {2}}{2}\right) ^ {1 / 2}} \exp \left\{- \frac {1}{8} \left(\mu_ {1} - \mu_ {2}\right) ^ {T} \left(\frac {\Sigma_ {1} + \Sigma_ {2}}{2}\right) ^ {- 1} \left(\mu_ {1} - \mu_ {2}\right) \right\} \tag {53}
$$
Applying it to distribution  $p_t'$  and  $q_t'$ , we have:
$$
\begin{array}{l} H ^ {2} \left(p _ {t} ^ {\prime}, q _ {t} ^ {\prime}\right) = 1 - \exp \left\{- \frac {1 - \bar {\alpha} _ {t}}{8 \left(1 - \bar {\alpha} _ {t - 1}\right) \beta_ {t}} \| \boldsymbol {\mu} _ {t, q} - \boldsymbol {\mu} _ {t, p} \| _ {2} ^ {2} \right\} (54) \\ \leq 1 - \exp \left\{- C _ {2} \left(C _ {1} \| \delta \| _ {2} + (\sqrt {L _ {u}} + 2 M) \sum_ {k = t + 1} ^ {T} \lambda (k, t) + \sum_ {k = t + 1} ^ {T} \lambda (k, t) \| s _ {\theta} \left(\tilde {\mathbf {x}} _ {k} ^ {\prime}, k\right) - \epsilon \left(\tilde {\mathbf {x}} _ {k} ^ {\prime}, k\right) \| _ {2}\right) ^ {2} \right\} (55) \\ \end{array}
$$
where  $C_1 = \left(\Pi_{s=t+1}^T \sqrt{\bar{\alpha}_s}\right) \sqrt{\bar{\alpha}_T}$  and  $C_2 = \frac{1 - \bar{\alpha}_t}{8(1 - \bar{\alpha}_{t-1})\beta_t}$ . Finally, it follows that: