SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Additionally assuming that  $b_{\mathrm{SDE}}$  and  $\Sigma$  are bounded, Yim et al. (Proposition 3.6, 2023) says that the distribution of  $\mathbf{Y}_t$  is  $G$ -invariant, and in particular  $\mathcal{L}(\bar{\mathbf{Y}}_0)$ .
D.3 Equivariant posterior maps
Theorem D.1 (Invariant prior stochastic process implies an equivariant posterior map). Using the language of Weiler et al. (2021) our tensor fields are sections of an associated vector bundle  $\mathcal{A}$  of a manifold  $M$  with a  $G$  structure. Let  $\operatorname{Isom}_{GM}$  be the group of  $G$ -structure preserving isometries on  $M$ . The action of this group on a section of the bundle  $f \in \Gamma(\mathcal{A})$  is given by
$$
\phi \triangleright f := \phi_ {*, \mathcal {A}} \circ f \circ \phi^ {- 1}
$$
(Weiler et al., 2021). Let  $f \sim P$ ,  $P$  a distribution over the space of section. Let  $\phi \triangleright P$  be the law of of  $\phi \triangleright f$ . Let  $\mu_x = \mathcal{L}(f(x)) = \pi_{x\#}P$ , the law of  $f$  evaluated at a point, where  $\pi_x$  is the canonical projection operator onto the marginal at  $x$ , # the pushforward operator in the measure theory sense,  $x \in M$  and  $y$  is in the fibre of the associated bundle. Let  $\mu_x^{x',y} = \mathcal{L}(f(x)|f(x') = y') = \pi_x\mu^{x',y'} = \pi_{x\#}\mathcal{L}(f|f(x') = y')$ , the conditional law of the process when given  $f(x') = y'$ .
Assume that the prior is invariant under the action of  $\operatorname{Isom}_{GM}$ , i.e. that
$$
\phi \triangleright \mu_ {x} = \left(\phi_ {*}, A\right) _ {\#} \mu_ {\phi^ {- 1} (x)} = \mu_ {x}
$$
Then the conditional measures are equivariant, in the sense that
$$
\phi \triangleright \mu_ {x} ^ {x ^ {\prime}, y ^ {\prime}} = (\phi_ {*, \mathcal {A}}) _ {\#} \mu_ {\phi^ {- 1} (x)} ^ {x ^ {\prime}, y ^ {\prime}} = \mu_ {x} ^ {\phi^ {- 1} (x), \phi_ {*, \mathcal {A}} (y)} = \mu_ {x} ^ {\phi \triangleright (x ^ {\prime}, y ^ {\prime})}
$$
Proof.  $\forall A, B$  test functions,  $\phi \in \operatorname{Isom}_{GM}$ ,
$$
\begin{array}{l} \operatorname {E} \left[ B \left(f \left(x ^ {\prime}\right)\right) A \left(\left(\phi \triangleright f\right) (x)\right) \right] = \operatorname {E} \left[ B \left(f \left(x ^ {\prime}\right)\right) A \left(\phi_ {*, \mathcal {A}} \circ f \circ \phi^ {- 1} (x)\right) \right] \\ = \mathrm {E} \left[ B \left(f \left(x ^ {\prime}\right)\right) \mathrm {E} \left[ A \left(\phi_ {*, \mathcal {A}} \left(F \left(\phi^ {- 1} (x)\right)\right) \mid F \left(x ^ {\prime}\right) \right] \right. \right] \\ = \operatorname {E} \left[ B \left(f \left(x ^ {\prime}\right)\right) \int A (y) \left(\phi_ {*}, \mathcal {A}\right) _ {\#} \mu_ {\phi^ {- 1} (x)} ^ {x ^ {\prime}, f \left(x ^ {\prime}\right)} (\mathrm {d} y) \right] \\ = \int B (y ^ {\prime}) \int A (y) \left(\phi_ {*}, \mathcal {A}\right) _ {\#} \mu_ {\phi^ {- 1} (x)} ^ {x ^ {\prime}, f \left(x ^ {\prime}\right)} (\mathrm {d} y) \mu_ {x ^ {\prime}} \left(\mathrm {d} y ^ {\prime}\right) \\ = \int B (y ^ {\prime}) \int A (y) \left(\phi \triangleright \mu_ {x} ^ {x ^ {\prime}, f \left(x ^ {\prime}\right)}\right) (\mathrm {d} y) \mu_ {x ^ {\prime}} (\mathrm {d} y ^ {\prime}) \\ \end{array}
$$
By invariance this quantity is also equal to
$$
\begin{array}{l} \operatorname {E} \left[ B \left(\left(\phi^ {- 1} \triangleright f\right) \left(x ^ {\prime}\right)\right) A \left(\left(\phi^ {- 1} \triangleright \phi \triangleright f\right) (x)\right) \right] = \operatorname {E} \left[ B \left(\left(\phi^ {- 1} \triangleright f\right) \left(x ^ {\prime}\right)\right) \operatorname {E} \left[ A \left(f (x)\right) \mid B \left(\left(\phi^ {- 1} \triangleright f\right) \left(x ^ {\prime}\right)\right) \right] \right] \\ = \mathrm {E} \left[ B \left(\phi_ {*}, A (f (\phi^ {- 1} (x ^ {\prime})))\right) \left[ A (F (x)) \mid \phi_ {*}, A (f (\phi^ {- 1} (x ^ {\prime}))) \right] \right] \\ = \operatorname {E} \left[ B \left(\tau_ {x ^ {\prime}, g} ^ {- 1} F (g x ^ {\prime})\right) \int A (y) \mu_ {x} ^ {\phi \left(x ^ {\prime}\right), \phi_ {* , \mathcal {A}} ^ {- 1} (y)} \right] (\mathrm {d} y) \\ = \int B (y ^ {\prime}) \int A (y) \mu_ {x} ^ {\phi \triangleright (x ^ {\prime}, y)} (\mathrm {d} y) \left(\phi_ {*, \mathcal {A}} ^ {- 1}\right) _ {\#} \mu_ {\phi (x ^ {\prime})} (\mathrm {d} y ^ {\prime}) \\ = \int B (y ^ {\prime}) \int A (y) \mu_ {x} ^ {\phi \triangleright (x ^ {\prime}, y)} (\mathrm {d} y) \left(\phi^ {- 1} \triangleright \mu_ {x ^ {\prime}}\right) (\mathrm {d} y ^ {\prime}) \\ \end{array}
$$
$$
\left(\phi \triangleright \mu_ {x} ^ {x ^ {\prime}, f \left(x ^ {\prime}\right)}\right) (\mathrm {d} y) \mu_ {x ^ {\prime}} (\mathrm {d} y ^ {\prime}) = \mu_ {x} ^ {\phi \triangleright \left(x ^ {\prime}, y\right)} (\mathrm {d} y) \left(\phi^ {- 1} \triangleright \mu_ {x ^ {\prime}}\right) (\mathrm {d} y ^ {\prime})
$$
By the stated invariance  $\phi^{-1} \triangleright \mu_{x'} = \mu_{x'}$ , hence
$$
\left(\phi \triangleright \mu_ {x} ^ {x ^ {\prime}, f \left(x ^ {\prime}\right)}\right) (\mathrm {d} y) = \mu_ {x} ^ {\phi \triangleright \left(x ^ {\prime}, y\right)} (\mathrm {d} y) a. e. y ^ {\prime}
$$
$$
\phi \triangleright \mu_ {x} ^ {x ^ {\prime}, f \left(x ^ {\prime}\right)} = \mu_ {x} ^ {\phi \triangleright \left(x ^ {\prime}, y\right)} \tag {68}
$$
E Langevin corrector and the iterative procedure of REPAINT (Lugmayr et al., 2022)
E.1 Langevin sampling scheme
Several previous schemes exist for conditional sampling from Diffusion models. Two different types of conditional sampling exist. Those that try to sample conditional on some part of the state space over which the diffusion model has been trained, such as in-painting or extrapolation tasks, and those that post-hoc attempt to condition on something outside the state space that the model has been trained on.
This first category is the one we are interested in, and in it we have:
- Replacement sampling (Song et al., 2021), where the reverse ODE or SDE is evolved but by fixing the conditioning data during the rollout. This method does produce visually coherent sampling in some cases, but is not an exact conditional sampling method.
- SMC-based methods (Trippe et al., 2022), which are an exact method up to the particle filter assumption. These can produce good results but can suffer from the usual SMC methods downsides on highly multi-model data such as particle diversity collapse.
- The RePaint scheme of (Lugmayr et al., 2022). While not originally proposed as an exact sampling scheme, we will show later that it can in fact be shown that this method is doing a specific instantiation of our newly proposed method, and is therefore exact.
- Amortisation methods, e.g. Phillips et al. (2022). While they can be effective, these methods can never perform exact conditional sampling, by definition.
Our goal is to produce an exact sampling scheme that does not rely on SMC-based methods. Instead, we base our method on Langevin dynamics. If we have a score function trained over the state space  $\pmb{x} = [\pmb{x}^c, \pmb{x}^*]$ , where  $\pmb{x}^c$  are the points we wish to condition on and  $\pmb{x}_s$  points we wish to sample, we exploit the following score breakdown:
$$
\nabla_ {\boldsymbol {x} ^ {*}} \log p (\boldsymbol {x} ^ {*} | \boldsymbol {x} ^ {c}) = \nabla_ {\boldsymbol {x} ^ {*}} \log p ([ \boldsymbol {x} ^ {*}, \boldsymbol {x} ^ {c} ]) - \nabla_ {\boldsymbol {x} ^ {*}} \log p (\boldsymbol {x} ^ {c}) = \nabla_ {\boldsymbol {x} ^ {*}} \log p (\boldsymbol {x})
$$
If we have access to the score on the joint variables, we, therefore, have access to the conditional score by simply only taking the gradient of the joint score for the variable we are not conditioning on.
Given we have learnt  $s_{\theta}(t, \boldsymbol{x}) \approx \nabla_{\boldsymbol{x}} \log p_t(\boldsymbol{x})$ , we could use this to perform Langevin dynamics at  $t = \epsilon$ , some time very close to 0. Similar to (Song and Ermon, 2019) however, this produces the twin issues of how to initialise the dynamics, given a random initialisation will start the sampler in a place where the score has been badly learnt, producing slow and inaccurate sampling.
Instead, we follow a scheme of tempered Langevin sampling detailed in Alg. 1. Starting at  $t = T$  we sample an initialisation of  $\pmb{y}^{*}$  based on the reference distribution. Progressing from  $t = T$  towards  $t = \epsilon$  we alternate between running a series of Lavgevin corrector steps to sample from the distribution  $p_{t,\pmb{x}^*}(\pmb{y}^*|\pmb{y}^c)$ , and a single backwards SDE step to sample from  $p_{\pmb{x}}(\pmb{y}_{t - \gamma}|\pmb{y}_t)$  with a step size  $\gamma$ . At each inner and outer step, we sample a noised version of the conditioning points  $\pmb{y}^c$  based forward SDE applying noise to these context points,  $p_{t,\pmb{x}^c}(\pmb{y}_t^c|\pmb{y}^c)$ . For the exactness of this scheme, all that matters is that at the end of the sampling scheme, we are sampling from  $p_{\pmb{x}^*}(\pmb{y}^*|\pmb{y}^c)$  (up to the  $\epsilon$  away from zero clipping of the SDE). The rest of the scheme is designed to map from the initial sample at  $t = T$  of  $\pmb{y}^*$  to a viable sample through regions where the score has been learnt well.
Given the noising scheme applied to the context points does not actually play into the theoretical exactness of the scheme, only the practical difficulty of staying near regions of well-learnt score, we could make a series of different choices for how to noise the context set at each step.
Figure 8: Comparison of different context noising schemes for the conditional sampling.
Table 4: Comparison of complexity of different noise sampling schemes for the context set.
Scheme	Closed-form noise	Simulated noise
Re-sample noise at every inner step	O(NI)	O(N2I2)
Re-sample noise at every outer step	O(N)	O(N2)
Sampling an SDE path on the context	O(N)	O(N)
No noise applied	-	-
The choices that present themselves are
1. The initial scheme of sampling context noise from the SDE every inner and outer step.
2. Only re-sampling the context noise every outer step, and keeping it fixed to this for each inner step associated with the outer step.
3. Instead of sampling independent marginal noise at each outer step, sampling a single noisig trajectory of the context set from the forward SDE and use this as the noise at each time.
4. Perform no noising at all. Effectively the replacement method with added Langevin sampling.
These are illustrated in Fig. 8. The main trade-off of different schemes is the speed at which the noise can be sampled vs sample diversity. In the Euclidean case, we have a closed form for the evolution of the marginal density of the context point under the forward SDE. In this case sampling the noise at a given time is  $\mathcal{O}(1)$  cost. On the other hand, in some instances such as nosing SDEs on general manifolds, we have to simulate this noise by discretising the forward SDE. In this case, it is  $\mathcal{O}(n)$  cost, where  $n$  is the number of discretisation steps in the SDE. For  $N$  outer steps and  $I$  inner steps, the complexity of the different noisings schemes is compared in Table 4. Note the conditional sampling scheme other than the noise sampling is  $\mathcal{O}(NI)$  complexity.
E.2 REPAINT (Lugmayr et al., 2022) correspondence
In this section, we show that:
Algorithm 1 Conditional sampling with Langevin dynamics.
Require: Score network  $\mathbf{s}_{\theta}(t,\pmb {x},\pmb {y})$  conditioning points  $(x^{c},y^{c})$  , query locations  $\pmb{x}^*$ $\bar{\pmb{x}} = [x^{c},x^{*}]$  Augmented inputs set   
 $\tilde{y}_T^*\sim \mathrm{N}(m(\pmb{x}^*),k(\pmb{x}^*,\pmb{x}^*))$  Sample initial noise   
for  $t\in \{T,T - \gamma ,\dots ,\epsilon \}$  do   
 $y_{t}^{c}\sim p_{t,x^{c}}(y_{t}^{c}|y_{0}^{c})$  Noise context outputs   
 $Z\sim \mathcal{N}(0,\mathrm{Id})$  Sample tangent noise   
 $[\underline{-},\tilde{y}_{t - \gamma}^{*}] = [\pmb{y}_{t}^{c},\pmb{y}_{t}^{*}] + \gamma \left\{-\frac{1}{2}\left(m(\bar{\pmb{x}}) - [\pmb{y}_{t}^{c},\pmb{y}_{t}^{*}]\right) + \mathrm{K}(\bar{\pmb{x}},\bar{\pmb{x}})\mathbf{s}_{\theta}(t,\bar{\pmb{x}},[\pmb{y}_{t}^{c},\tilde{\pmb{y}}_{t}^{*}])\right\} +\sqrt{\gamma}\mathrm{K}(\bar{\pmb{x}},\bar{\pmb{x}})^{1 / 2}Z\triangleright$  Euler-Maruyama step   
for  $l\in \{1,\ldots ,L\}$  do   
 $y_{t - \gamma}^{c}\sim p_{t - \gamma ,x^{c}}(y_{t - \gamma}^{c}|y_{0}^{c})$  Noise context outputs   
 $Z\sim \mathcal{N}(0,\mathrm{Id})$  Sample tangent noise   
 $[\underline{-},\tilde{y}_{t - \gamma}^{*}] = [\underline{-},\tilde{y}_{t - \gamma}^{*}] + \frac{\gamma}{2}\mathrm{K}(\bar{\pmb{x}},\bar{\pmb{x}})\mathbf{s}_{\theta}(t - \gamma ,\bar{\pmb{x}},[\pmb{y}_{t - \gamma}^{c},\tilde{\pmb{y}}_{t - \gamma}^{*}]) + \sqrt{\gamma}\mathrm{K}(\bar{\pmb{x}},\bar{\pmb{x}})^{1 / 2}Z\triangleright$  Langevin step   
 $y_{t - \gamma}^{*} = \tilde{y}_{t - \gamma}^{*}$    
return  $y_{\epsilon}^{*}$
(a) Alg. 1 and Alg. 2 Repaint from (Lugmayr et al., 2022) are equivalent in a specific setting.
(b) There exists a continuous limit (SDE) for both procedures. This SDE targets a probability density which does not correspond to  $p(x_{t_0} | x_0^c)$ .
(c) When  $t_0 \to 0$  this probability measure converges to  $p(x_0|x_0^c)$  which ensures the correctness of the proposed sampling scheme.
We begin by recalling the conditional sampling algorithm we study in Alg. 1 and Alg. 2.
Algorithm 2 REPAINT (Lugmayr et al., 2022).
Require: Score network  $\mathbf{s}_{\theta}(t,\pmb {x},\pmb {y})$  conditioning points  $(x^{c},y^{c})$  query locations  $\pmb{x}^*$ $\bar{\pmb{x}} = [x^{c},x^{*}]$  Augmented inputs set   
 $[\pmb{y}_T^c,\pmb{y}_T^* ]\sim \mathrm{N}(m(\bar{\pmb{x}}),k(\bar{\pmb{x}},\bar{\pmb{x}}))$  Sample initial noise   
for  $t\in \{T,T - \gamma ,\dots ,\epsilon \}$  do   
 $\tilde{\pmb{y}}_t^* = \pmb {y}_t^*$    
for  $l\in \{1,\ldots ,L\}$  do   
 $y_{t}^{c}\sim \mathrm{N}(mt(x^{c};y^{c}),kt(x^{c},x^{c};y^{c}))$  Noise context outputs   
 $Z\sim \mathrm{N}(0,\mathrm{Id})$  □ Sample tangent noise   
 $[,-,\tilde{y}_{t - \gamma}^{*}] = [y_{t}^{c},\tilde{y}_{t}^{*}] + \gamma \left\{-\frac{1}{2}\left(m(\bar{\pmb{x}}) - [y_{t}^{c},\tilde{y}_{t}^{*}]\right) + \mathrm{K}(\bar{\pmb{x}},\bar{\pmb{x}})\mathbf{s}_{\theta}(t,\bar{\pmb{x}},[y_{t}^{c},\tilde{y}_{t}^{*}])\right\} +\sqrt{\gamma}\mathrm{K}(\bar{\pmb{x}},\bar{\pmb{x}})^{1 / 2}Z$ $\triangleright$  Reverse step   
 $Z\sim \mathrm{N}(0,\mathrm{Id})$  □ Sample tangent noise   
 $\tilde{\pmb{y}}_t^* = \tilde{\pmb{y}}_{t - \gamma}^* +\gamma \left\{\frac{1}{2}\left(m(\pmb{x}^*) - \tilde{\pmb{y}}_{t - \gamma}^*\right)\right\} +\sqrt{\gamma}\mathrm{K}(\pmb{x}^*,\pmb{x}^*)^{1 / 2}Z$  □ Forward step   
 ${\pmb{y}}_{t - \gamma}^{*} = {\pmb{\tilde{y}}}_{t - \gamma}^{*}$
First, we start by describing the RePaint algorithm (Lugmayr et al., 2022). We consider  $(Z_k^0, Z_k^1, Z_k^2)_{k \in \mathbb{N}}$  a sequence of independent Gaussian random variables such that for any  $k \in \mathbb{N}$ ,  $Z_k^1$  and  $Z_k^2$  are  $d$ -dimensional Gaussian random variables with zero mean and identity covariance matrix and  $Z_k^0$  is a  $p$ -dimensional Gaussian random variable with zero mean and identity covariance matrix. We assume that the whole sequence to be inferred is of size  $d$  while the context is of size  $p$ . For simplicity, we only consider the Euclidean setting with  $\mathrm{K} = \mathrm{Id}$ . The proofs can be adapted to cover the case  $\mathrm{K} \neq \mathrm{Id}$  without loss of generality.
Let us fix a time  $t_0 \in [0, T]$ . We consider the chain  $(X_k)_{k \in \mathbb{N}}$  given by  $X_0 \in \mathbb{R}^d$  and for any  $k \in \mathbb{N}$ , we define
$$
X _ {k + 1 / 2} = \mathrm {e} ^ {\gamma} X _ {k} + 2 \left(\mathrm {e} ^ {\gamma} - 1\right) \nabla_ {x _ {k}} \log p _ {t _ {0}} \left(\left[ X _ {k}, X _ {k} ^ {c} \right]\right) + \left(\mathrm {e} ^ {2 \gamma} - 1\right) ^ {1 / 2} Z _ {k} ^ {1}, \tag {69}
$$
where  $X_{k}^{c} = \mathrm{e}^{-t_{0}}X_{0}^{c} + (1 - \mathrm{e}^{-2t_{0}})^{1 / 2}Z_{k}^{0}$ . Finally, we consider
$$
X _ {k + 1} = \mathrm {e} ^ {- \gamma} X _ {k + 1 / 2} + \left(1 - \mathrm {e} ^ {- 2 \gamma}\right) ^ {1 / 2} Z _ {k} ^ {2}. \tag {70}
$$
Note that (69) corresponds to one step of backward SDE integration and (70) corresponds to one step of forward SDE integration. In both cases we have used the exponential integrator, see (De
Bortoli, 2022) for instance. While we use the exponential integrator in the proofs for simplicity other integrators such as the classical Euler-Maruyama integration could have been used. Combining (69) and (70), we get that for any  $k \in \mathbb{N}$  we have
$$
X _ {k + 1} = X _ {k} + 2 \left(1 - \mathrm {e} ^ {- \gamma}\right) \nabla_ {x _ {k}} \log p _ {t _ {0}} \left(\left[ X _ {k}, X _ {k} ^ {c} \right]\right) + \left(1 - \mathrm {e} ^ {- 2 \gamma}\right) ^ {1 / 2} \left(Z _ {k} ^ {1} + Z _ {k} ^ {2}\right). \tag {71}
$$
Remarking that  $(Z_{k})_{k\in \mathbb{N}} = ((Z_{k}^{1} + Z_{k}^{3}) / \sqrt{2})_{k\in \mathbb{N}}$  is a family of  $d$ -dimensional Gaussian random variables with zero mean and identity covariance matrix, we get that for any  $k\in \mathbb{N}$
$$
X _ {k + 1} = X _ {k} + 2 \left(1 - \mathrm {e} ^ {- \gamma}\right) \nabla_ {x _ {k}} \log p _ {t _ {0}} \left(\left[ X _ {k}, X _ {k} ^ {c} \right]\right) + \sqrt {2} \left(1 - \mathrm {e} ^ {- 2 \gamma}\right) ^ {1 / 2} Z _ {k}, \tag {72}
$$
where we recall that  $X_{k}^{c} = \mathrm{e}^{-t_{0}}X_{0}^{c} + (1 - \mathrm{e}^{-2t_{0}})^{1 / 2}Z_{k}^{0}$ . Note that the process (72) is another version of the Repaint algorithm (Lugmayr et al., 2022), where we have concatenated the denoising and noising procedure. With this formulation, it is clear that Repaint is equivalent to Alg. 1. In what follows, we identify the limiting SDE of this process.
In what follows, we describe the limiting behavior of (72) under mild assumptions on the target distribution. In what follows, for any  $x_{t_0} \in \mathbb{R}^d$ , we denote
$$
b \left(x _ {t _ {0}}\right) = 2 \int_ {\mathbb {R} ^ {p}} \nabla_ {x _ {t _ {0}}} \log p _ {t _ {0}} \left(\left[ x _ {t _ {0}}, x _ {t _ {0}} ^ {c} \right]\right) p _ {t _ {0} | 0} \left(x _ {t _ {0}} ^ {c} \mid x _ {0} ^ {c}\right) \mathrm {d} x _ {t _ {0}} ^ {c}. \tag {73}
$$
We emphasize that  $b / 2 \neq \nabla_{x_{t_0}} \log p(\cdot | x_0^c)$ . In particular, using Tweedie's identity, we have that for any  $x_{t_0} \in \mathbb{R}^d$
$$
\nabla \log p _ {t _ {0}} \left(x _ {t _ {0}} \mid x _ {0} ^ {c}\right) = \int_ {\mathbb {R} ^ {p}} \nabla_ {x _ {t _ {0}}} \log p \left(\left[ x _ {t _ {0}}, x _ {t _ {0}} ^ {c} \right] \mid x _ {0} ^ {c}\right) p \left(x _ {t _ {0}} ^ {c} \mid x _ {t _ {0}}, x _ {0} ^ {c}\right) \mathrm {d} x _ {t _ {0}} ^ {c}. \tag {74}
$$
We introduce the following assumption.
Assumption 1. There exist  $\mathsf{L}, C \geq 0, \mathfrak{m} > 0$  such that for any  $x_{t_0}^c, y_t^c \in \mathbb{R}^p$  and  $x_{t_0}, y_t \in \mathbb{R}^d$
$$
\left\| \nabla \log p _ {t _ {0}} \left(\left[ x _ {t _ {0}}, x _ {t _ {0}} ^ {c} \right]\right) - \nabla \log p _ {t _ {0}} \left(\left[ y _ {t}, y _ {t} ^ {c} \right]\right) \right\| \leq \mathrm {L} \left(\left\| x _ {t _ {0}} - y _ {t} \right\| + \left\| x _ {t _ {0}} ^ {c} - y _ {t} ^ {c} \right\|\right). \tag {75}
$$
Assumption 1 ensures that there exists a unique strong solution to the SDE associated with (72). Note that conditions under which  $\log p_{t_0}$  is Lipschitz are studied in De Bortoli (2022). In the theoretical literature on diffusion models the Lipschitzness assumption is classical, see Lee et al. (2023) and Chen et al. (2022).
We denote  $((\mathbf{X}_t^\gamma)_{t\geq 0})_{\gamma >0}$  the family of processes such that for any  $k\in \mathbb{N}$  and  $\gamma >0$  , we have for any  $t\in [k\gamma ,(k + 1)\gamma)$ $\mathbf{X}_t^\gamma = (1 - (t - k\gamma) / \gamma)\mathbf{X}_{k\gamma}^\gamma +(t - k\gamma) / \gamma \mathbf{X}_{(k + 1)\gamma}^\gamma$  and
$$
\mathbf {X} _ {(k + 1) \gamma} ^ {\gamma} = \mathbf {X} _ {k \gamma} ^ {\gamma} + 2 (1 - \mathrm {e} ^ {- \gamma}) \nabla_ {\mathbf {X} _ {k \gamma} ^ {\gamma}} \log p _ {t _ {0}} ([ \mathbf {X} _ {k \gamma} ^ {\gamma}, \mathbf {X} _ {k \gamma} ^ {c, n} ]) + \sqrt {2} (1 - \mathrm {e} ^ {- 2 \gamma}) ^ {1 / 2} \mathbf {Z} _ {k \gamma} ^ {\gamma}, \tag {76}
$$
where  $(\mathbf{Z}_{k\gamma}^{\gamma})_{k\in \mathbb{N},\gamma >0}$  is a family of independent  $d$  -dimensional Gaussian random variables with zero mean and identity covariance matrix and for any  $k\in \mathbb{N},\gamma >0,\mathbf{X}_{k\gamma}^{c,\gamma} = \mathrm{e}^{-t_0}x_0^c +(1 - \mathrm{e}^{-2t_0})^{1 / 2}\mathbf{Z}_{k\gamma}^{0,\gamma}$ , where  $(\mathbf{Z}_{k\gamma}^{0,\gamma})_{k\in \mathbb{N},\gamma >0}$  is a family of independent  $p$  -dimensional Gaussian random variables with zero mean and identity covariance matrix. This is a linear interpolation of the Repaint algorithm in the form of (72).
Finally, we denote  $(\mathbf{X}_t)_{t\geq 0}$  such that
$$
\mathrm {d} \mathbf {X} _ {t} = b \left(\mathbf {X} _ {t}\right) \mathrm {d} t + 2 \mathbf {B} _ {t}, \quad \mathbf {X} _ {0} = x _ {0}. \tag {77}
$$
We recall that  $b$  depends on  $t_0$  but  $t_0$  is fixed here. This means that we are at time  $t_0$  in the diffusion and consider a corrector at this stage. The variable  $t$  does not correspond to the backward evolution but to the forward evolution in the corrector stage. Under Assumption 1, (77) admits a unique strong solution. The rest of the section is dedicated to the proof of the following result.
Theorem E.1. Assume Assumption 1. Then  $\lim_{n\to +\infty}(\mathbf{X}_t^{1 / n})_{t\geq 0} = (\mathbf{X}_t)_{t\geq 0}$ .
This result is an application of Stroock and Varadhan (2007, Theorem 11.2.3). It explicits what is the continuous limit of the Repaint algorithm (Lugmayr et al., 2022).
In what follows, we verify that the assumptions of this result hold in our setting. For any  $\gamma > 0$  and  $x \in \mathbb{R}^d$ , we define
$$
\begin{array}{l} b _ {\gamma} (x) = (2 / \gamma) \left[ \left(1 - \mathrm {e} ^ {- \gamma}\right) \int_ {\mathbb {R} ^ {d}} \nabla_ {x _ {t _ {0}}} \log p _ {t _ {0}} \left(\left[ x _ {t _ {0}}, x _ {t _ {0}} ^ {c} \right]\right) p _ {t _ {0} | 0} \left(x _ {t _ {0}} ^ {c} \mid x _ {0} ^ {c}\right) \mathrm {d} x _ {t _ {0}} ^ {c} \right. (78) \\ - (1 / \gamma) \mathbb {E} \left[ \left(\mathbf {X} _ {(k + 1) \gamma} ^ {\gamma} - \mathbf {X} _ {k \gamma} ^ {\gamma}\right) \mathbf {1} _ {\| \mathbf {X} _ {(k + 1) \gamma} ^ {\gamma} - \mathbf {X} _ {k \gamma} ^ {\gamma} \| \geq 1} \mid \mathbf {X} _ {k \gamma} = x \right], (79) \\ \end{array}
$$
$$
\begin{array}{l} \Sigma_ {\gamma} (x) = (4 / \gamma) (1 - \mathrm {e} ^ {- \gamma}) ^ {2} \int_ {\mathbb {R} ^ {d}} \nabla_ {x _ {t _ {0}}} \log p _ {t _ {0}} ([ x _ {t _ {0}}, x _ {t _ {0}} ^ {c} ]) ^ {\otimes 2} p _ {t _ {0} | 0} (x _ {t _ {0}} ^ {c} | x _ {0} ^ {c}) \mathrm {d} x _ {t _ {0}} ^ {c} + (2 / \gamma) (1 - \mathrm {e} ^ {- 2 \gamma}) \text {I d} (80) \\ - (1 / \gamma) \mathbb {E} \left[ \left(\mathbf {X} _ {(k + 1) \gamma} ^ {\gamma} - \mathbf {X} _ {k \gamma} ^ {\gamma}\right) ^ {\otimes 2} \mathbf {1} _ {\| \mathbf {X} _ {(k + 1) \gamma} ^ {\gamma} - \mathbf {X} _ {k \gamma} ^ {\gamma} \| \geq 1} \mid \mathbf {X} _ {k \gamma} = x \right]. (81) \\ \end{array}
$$
Note that for any  $\gamma > 0$  and  $x \in \mathbb{R}^d$ , we have
$$
b _ {\gamma} (x) = \mathbb {E} \left[ \mathbf {1} _ {\| \mathbf {X} _ {(k + 1) \gamma} ^ {\gamma} - \mathbf {X} _ {k \gamma} ^ {\gamma} \| \leq 1} \left(\mathbf {X} _ {(k + 1) \gamma} ^ {\gamma} - \mathbf {X} _ {k \gamma} ^ {\gamma}\right) \mid \mathbf {X} _ {k \gamma} ^ {\gamma} = x \right] \tag {82}
$$
$$
\Sigma_ {\gamma} (x) = \mathbb {E} \left[ \mathbf {1} _ {\| \mathbf {X} _ {(k + 1) \gamma} ^ {\gamma} - \mathbf {X} _ {k \gamma} ^ {\gamma} \| \leq 1} \left(\mathbf {X} _ {(k + 1) \gamma} ^ {\gamma} - \mathbf {X} _ {k \gamma} ^ {\gamma}\right) ^ {\otimes 2} \mid \mathbf {X} _ {k \gamma} ^ {\gamma} = x \right] \tag {83}
$$
Lemma E.2. Assume Assumption 1. Then, we have that for any  $R,\varepsilon >0$  and  $\gamma \in (0,1)$
$$
\lim  _ {\gamma \rightarrow 0} \sup  \left\{\| \Sigma_ {\gamma} (x) - \Sigma (x) \| \mid x \in \mathbb {R} ^ {d}, \| x \| \leq R \right\} = 0, \tag {85}
$$