SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\alpha_{t} := 1 - \beta_{t}$ ,  $\bar{\alpha}_{t} := \prod_{s=1}^{t} \alpha_{s}$  and  $\pmb{\mu}_{\theta}(\mathbf{x}_t, t) = \frac{1}{\sqrt{\alpha_t}} \left( \mathbf{x}_t - \frac{\beta_t}{\sqrt{1 - \bar{\alpha}_t}} \pmb{\epsilon}_{\theta}(\mathbf{x}_t, t) \right)$ . Intuitively, this loss says that  $\pmb{\epsilon}_{\theta}(\mathbf{x}_t, t)$  should fit the noise  $\pmb{\epsilon}$  added to the initial datapoint  $\mathbf{x}_0$  up to time  $t$ . For a full account of this derivation, we refer the reader to Ho et al. [21].
A.2.1 Classifier Guidance
Sohl-Dickstein et al. [61] also propose a method for steering the sampling process of diffusion models. Intuitively, if the original diffusion model computes a distribution  $p(\mathbf{x}_0)$ , we would like to make it compute  $p(\mathbf{x}_0|y)$ , where we assume, for the sake of concreteness, that  $y \in \{0, 1\}$ . Applying Bayes's rule gives the factorization  $p(\mathbf{x}_0|y) \propto p(\mathbf{x}_0) \cdot p(y|\mathbf{x}_0)$ . The first term is the original distribution the diffusion model is trained on. The second term is a probabilistic classifier, inferring the probability that  $y = 1$  for a given sample  $\mathbf{x}_0$ .
(a) Probabilistic Graphical Model (PGM) for sampling from a Markov Decision Process (MDP) with a fixed policy  $\pi$ .
(b) PGM for planning, with optimality variables  $\mathcal{O}_{1:T}$ .
Figure 5: Reinforcement Learning as Probabilistic Inference (see App. B for a detailed explanation).
We now leverage the framework in A.1 to derive the classifier guidance procedure. Equation 16 shows we can sample from the backward process by modeling  $\nabla_{\mathbf{x}}\log p_t(\mathbf{x}_t)$  using  $\mathbf{s}_{\Theta}(\mathbf{x}_t,t)$ . Thus, to model  $p(\mathbf{x}_0|y)$ , we may instead approximate  $\nabla_{\mathbf{x}}\log p(\mathbf{x}_t|y)$  using:
$$
\begin{array}{l} \nabla_ {\mathbf {x}} \log p (\mathbf {x} _ {t} | y) = \nabla_ {\mathbf {x}} \log \frac {p (\mathbf {x} _ {t}) p (y | \mathbf {x} _ {t})}{p (y)} (19) \\ = \nabla_ {\mathbf {x}} \log p (\mathbf {x} _ {t}) + \nabla_ {\mathbf {x}} \log p (y | \mathbf {x} _ {t}) (20) \\ \approx \mathbf {s} _ {\Theta} \left(\mathbf {x} _ {t}, t\right) + \nabla_ {\mathbf {x}} \rho \left(\mathbf {x} _ {t}, t\right) (21) \\ \end{array}
$$
where  $\rho (\mathbf{x},t)$  is a neural network approximating  $\log p(y|\mathbf{x}_t)$ . In practice, we multiply the gradients of  $\rho$  by a small constant  $\omega$ , called the guidance scale.
The guided reverse SDE hence becomes:
$$
d \bar {\mathbf {x}} _ {t} = \left[ \mathbf {f} \left(\bar {\mathbf {x}} _ {t}, t\right) - g (t) ^ {2} \left[ \mathbf {s} _ {\Theta} \left(\bar {\mathbf {x}} _ {t}, t\right) + \omega \nabla_ {\mathbf {x}} \rho \left(\bar {\mathbf {x}} _ {t}, t\right) \right] \right] d t + g (t) d \bar {\mathbf {w}} _ {t} \tag {22}
$$
The main takeaway is, informally, that we can steer a diffusion model to produce samples with some property  $y$  by gradually pushing the samples in the direction that maximizes the output of a classifier predicting  $p(y|\mathbf{x})$ .
Appendix B More Details on Reinforcement Learning as Probabilistic Inference
Levine [37] provides an in-depth exposition on existing methods for approaching sequential decision-making from a probabilistic and causal angle, referencing the work of e.g. [66, 70, 71, 31, 30]. We refer to it as the Control as Inference Framework (CIF).
The starting point for the formulation is the classical notion of a Markov Decision Problem (MDP), a stochastic process given by a state space  $S$ , an action space  $\mathcal{A}$ , a (possibly stochastic) transition function  $\mathcal{T}: S \times \mathcal{A} \to \Delta_S$ , and a reward function  $r: S \times \mathcal{A} \to \Delta_{\mathbb{R}_{\geq 0}}$ . Here  $\Delta_{\mathcal{X}}$  denotes the set of distributions over a set  $\mathcal{X}$ . In this context, a policy  $\pi: S \to \Delta_A$  represents a (possibly stochastic) way of choosing an action given the current state.
The MDP starts at an initial state  $s_0 \in S$ , and evolves by sampling  $a_t \sim \pi(s_t)$ , and then  $s_{t+1} \sim \mathcal{T}(s_t, a_t)$  for  $t \geq 0$ . The reward received at time  $t$  is  $r_t \sim r(s_t, a_t)$ . We consider here an episodic setting, where the MDP stops at a fixed time  $T > 0$ . We call the sequence  $((s_t, a_t))_{t=0}^T$  a trajectory. Sampling corresponds to the Probabilistic Graphical Model (PGM) in 5a, where each action depends on the current state, and the next state depends on the current state and the current action.
To encode the process of choosing a policy to optimize a reward, this PGM can be extended with optimality variables, as per Fig. 5b. They are defined as  $\mathcal{O}_t\sim \mathrm{Ber}(e^{-r(st,a_t)})$ , so that their distribution encodes the dependency of reward on current states and actions. The problem of optimizing the reward can then be recast as sampling from  $p(\tau |\mathcal{O}_{1:T})$ . Following [37], one can apply Bayes's Rule to obtain
$$
p (\tau | \mathcal {O} _ {1: T}) \propto p (\tau) \cdot p (\mathcal {O} _ {1: T} | \tau) \tag {23}
$$
which factorizes the distribution of optimal trajectories (up to a normalizing constant) as a prior  $p(\tau)$  over trajectories and a likelihood term  $p(\mathcal{O}_{1:T}|\tau)$ . Observe that, from the definition of  $\mathcal{O}_{1:T}$  and the PGM factorization in Fig. 5b, we have  $p(\mathcal{O}_{1:T}|\tau) = e^{-\sum_t r(s_t,a_t)}$ . Hence, in the CIF,  $-\log p(\mathcal{O}_{1:T}|\tau)$  (a negative log likelihood) corresponds to the cumulative reward of a trajectory  $\tau$ .
Appendix C Details on Main Derivations
C.1 Existence and Uniqueness of Strong Solutions to Stochastic Differential Equations
Theorem C.1 (Existence and uniqueness theorem for SDEs, c.f. page 66 of Øksendal and Øksendal [45]). Let  $T > 0$  and  $b(\cdot, \cdot): [0, T] \times \mathbb{R}^n \to \mathbb{R}^n$ ,  $\sigma(\cdot, \cdot): [0, T] \times \mathbb{R}^n \to \mathbb{R}^{n \times m}$  be measurable functions satisfying
$$
| b (t, x) | + | \sigma (t, x) | \leq C (1 + | x |); \quad x \in \mathbb {R} ^ {n}, t \in [ 0, T ]
$$
for some constant  $C$ , where  $|\sigma|^2 = \sum |\sigma_{ij}|^2$ , and such that
$$
| b (t, x) - b (t, y) | + | \sigma (t, x) - \sigma (t, y) | \leq D | x - y |; \quad x, y \in \mathbb {R} ^ {n}, t \in [ 0, T ]
$$
for some constant  $D$ . Let  $Z$  be a random variable which is independent of the  $\sigma$ -algebra  $\mathcal{F}_{\infty}^{(m)}$  generated by  $B_{s}(\cdot), s \geq 0$  and such that
$$
\mathbb {E} \left[ | Z | ^ {2} \right] <   \infty
$$
Then the stochastic differential equation
$$
d X _ {t} = b (t, X _ {t}) d t + \sigma (t, X _ {t}) d B _ {t}, \quad 0 \leq t \leq T, X _ {0} = Z
$$
has a unique  $t$ -continuous solution  $X_{t}(\omega)$  with the property that  $X_{t}(\omega)$  is adapted to the filtration  $\mathcal{F}_t^Z$  generated by  $Z$  and  $B_{s}(\cdot); s \leq t$  and
$$
\mathbb {E} \left[ \int_ {0} ^ {T} | X _ {t} | ^ {2} d t \right] <   \infty .
$$
Remark C.2. In the above, the symbol  $|\cdot|$  is overloaded, and taken to mean the norm of its argument. As everything in sight is finite-dimensional, and as all norms in finite-dimensional normed vector spaces are topologically equivalent (Theorem 2.4-5 in Kreyszig [34]), the stated growth conditions do not depend on the particular norm chosen for  $\mathbb{R}^n$ .
Also, whenever we talk about a solution to an SDE with respect to e.g. the standard Brownian motion  $\mathbf{w}$  and an initial condition  $\mathbf{z}$ , we are always referring only to  $(\mathcal{F}_t^{\mathbf{z}})_{t\geq 0}$ -adapted stochastic processes, as per Theorem C.1.
Corollary C.3. Fix a Brownian Motion  $\mathbf{w}$  and let  $(\mathcal{F}_t)_{t\geq 0}$  be its natural filtration. Consider an SDE  $d\mathbf{x}_t = \mathbf{f}(\mathbf{x}_t,t) + g(t)d\mathbf{w}_t$  with initial condition  $\mathbf{z}\in \mathcal{L}^2$  independent of  $\mathcal{F}_{\infty} = \cup_{t\geq 0}\mathcal{F}_t$ . Suppose  $\mathbf{f}:\mathbb{R}^n\times [0,T]\to \mathbb{R}^n$  is Lipschitz with respect to  $(\mathbb{R}^n\times [0,T],\mathbb{R}^n)$  and  $g:[0,T]\to \mathbb{R}_{\geq 0}$  is continuous and bounded.
Then the conclusion of C.1 holds and there is an a.s. unique  $(\mathcal{F}_t^{\mathbf{z}})_{t\geq 0}$ -adapted solution  $(\mathbf{x}_t)_{t\geq 0}$  having a.s. continuous paths, where  $(\mathcal{F}_t^{\mathbf{z}})_{t\geq 0}$  is defined as in C.1.
Proof. Firstly, note that, as  $\mathbf{f}$  and  $g$  are continuous, they are Lebesgue-measurable. It remains to check the growth conditions in C.1.
As  $\mathbf{f}$  is Lipschitz, there exists  $C_1$  (w.l.o.g.  $C_1 > 1$ ) such that, for any  $\mathbf{x}, \mathbf{y} \in \mathbb{R}^n$  and  $t, s \in [0, T]$ :
$$
\left| \mathbf {f} (\mathbf {x}, t) - \mathbf {f} (\mathbf {y}, s) \right| \leq C _ {1} \left(\left| \mathbf {x} - \mathbf {y} \right| + \left| t - s \right|\right) \tag {24}
$$
$$
\leq C _ {1} (| \mathbf {x} - \mathbf {y} | + T) \tag {25}
$$
In particular, taking  $\mathbf{y} = \mathbf{0}$  and  $s = 0$ , we obtain that  $|\mathbf{f}(\mathbf{x},t) - \mathbf{f}(\mathbf{0},0)|\leq C_1(|\mathbf{x}| + T)$ . Let  $M = |\mathbf{f}(\mathbf{0},0)| / C_{1}$ . Then, by the triangle inequality,  $|\mathbf{f}(\mathbf{x},t)|\leq C_1(|\mathbf{x}| + T + M)$ .
As  $g$  is bounded, there exists  $C_2 > 0$  such that  $|g(t)| \leq C_2$ , and so for any  $t, s \in [0,T]$ , we have:
$$
\left| g (t) - g (s) \right| \leq 2 C _ {2} \tag {26}
$$
Hence we have, for  $t\in [0,T]$  and  $\mathbf{x},\mathbf{y}\in \mathbb{R}^n$  ..
$$
\left| \mathbf {f} (\mathbf {x}, t) \right| + | g (t) | \leq \left((T + M) C _ {1} + C _ {2}\right) (1 + | \mathbf {x} |) \quad \left| \mathbf {f} (\mathbf {x}, t) - \mathbf {f} (\mathbf {y}, t) \right| \leq C _ {1} | \mathbf {x} - \mathbf {y} | \tag {27}
$$
which yields the growth conditions required in C.1, and the conclusion follows.
C.2 Proof of Theorem 4.1
Proof of Theorem 4.1. We reference the fundamental result on the existence and uniqueness of strong solutions to SDEs, presented in Section 5.2.1 of [45] and reproduced above. In particular, Corollary C.3 shows that the Existence and Uniqueness Theorem C.1 applies for SDEs 6 and 7, by our assumptions on  $\mathbf{f}^{(1)}$ ,  $\mathbf{f}^{(2)}$  and  $g$ . As also we restrict our attention to  $\mathbf{h}$  that are Lipschitz, and sums of Lipschitz functions are also Lipschitz, the corollary also applies to SDE 8. This establishes the existence of a.s. unique (adapted) solutions with a.s. continuous sample paths for these SDEs. Call them  $\mathbf{x}^{(1)}$ ,  $\mathbf{x}^{(2)}$  and  $\mathbf{x}$ , respectively.
Denote  $\mathbf{h}_1(\mathbf{x},t) = \mathbf{f}^{(2)}(\mathbf{x},t) - \mathbf{f}^{(1)}(\mathbf{x},t)$ . The choice  $\mathbf{h} = \mathbf{h}_1$  makes SDE 8 have the same drift and noise coefficients as SDE 7, and so, by the uniqueness of solutions we established, it follows that  $\mathbf{x}_t^{(2)} = \mathbf{x}_t$  for all  $t$  a.s., with this choice of  $\mathbf{h}$ .
Now suppose we have a Lipschitz-continuous  $\tilde{\mathbf{h}}$  which yields an a.s. unique,  $t$ -continuous solution  $(\tilde{\mathbf{x}}_t)_{t\geq 0}$  that is indistinguishable from  $\mathbf{x}^{(2)}$ . (i.e. equal to  $\mathbf{x}^{(2)}$  for all  $t$ , a.s.). We show that  $\tilde{\mathbf{h}} = \mathbf{h}_1$ .
As  $\tilde{\mathbf{x}}$  and  $\mathbf{x}^{(2)}$  are indistinguishable and satisfy SDEs with the same noise coefficient, we obtain a.s., for any  $t\in [0,T]$ :
$$
\begin{array}{l} 0 = \tilde {\mathbf {x}} _ {t} - \mathbf {x} _ {t} ^ {(2)} (28) \\ = \left(\int_ {0} ^ {t} \mathbf {f} ^ {(1)} \left(\tilde {\mathbf {x}} _ {s}, s\right) + \tilde {\mathbf {h}} \left(\tilde {\mathbf {x}} _ {s}, s\right) d s + \int_ {0} ^ {t} g (s) d \mathbf {w} _ {s}\right) - \left(\int_ {0} ^ {t} \mathbf {f} ^ {(2)} \left(\mathbf {x} _ {s} ^ {(2)}, s\right) d s + \int_ {0} ^ {t} g (s) d \mathbf {w} _ {s}\right) (29) \\ = \int_ {0} ^ {t} \mathbf {f} ^ {(1)} \left(\tilde {\mathbf {x}} _ {s}, s\right) + \tilde {\mathbf {h}} \left(\tilde {\mathbf {x}} _ {s}, s\right) d s - \int_ {0} ^ {t} \mathbf {f} ^ {(2)} \left(\mathbf {x} _ {s} ^ {(2)}, s\right) d s (30) \\ = \int_ {0} ^ {t} \mathbf {f} ^ {(1)} \left(\mathbf {x} _ {s} ^ {(2)}, s\right) + \tilde {\mathbf {h}} \left(\mathbf {x} _ {s} ^ {(2)}, s\right) d s - \int_ {0} ^ {t} \mathbf {f} ^ {(2)} \left(\mathbf {x} _ {s} ^ {(2)}, s\right) d s (31) \\ = \int_ {0} ^ {t} \tilde {\mathbf {h}} \left(\mathbf {x} _ {s} ^ {(2)}, s\right) - \left(\mathbf {f} ^ {(2)} \left(\mathbf {x} _ {s} ^ {(2)}, s\right) - \mathbf {f} ^ {(1)} \left(\mathbf {x} _ {s} ^ {(2)}, s\right)\right) d s (32) \\ = \int_ {0} ^ {t} \tilde {\mathbf {h}} \left(\mathbf {x} _ {s} ^ {(2)}, s\right) - \mathbf {h} _ {1} \left(\mathbf {x} _ {s} ^ {(2)}, s\right) d s (33) \\ \end{array}
$$
where in 29 we substitute the integral forms of the SDEs satisfied by  $\tilde{\mathbf{x}}$  and  $\mathbf{x}^{(2)}$ , and in 31 we use that the processes are indistinguishable to replace  $\tilde{\mathbf{x}}$  by  $\mathbf{x}^{(2)}$  in one of the integrals.
Hence, with probability  $1$ ,  $\int_0^t\tilde{\mathbf{h}} (\mathbf{x}_s^{(2)},s)ds = \int_0^t\mathbf{h}_1(\mathbf{x}_s^{(2)},s)ds$ . As  $\mathbf{x}^{(2)}$  is a.s. continuous, we also have that the mappings  $s\mapsto \mathbf{h}_1(\mathbf{x}_s^{(2)},s)$  and  $s\mapsto \tilde{\mathbf{h}} (\mathbf{x}_s^{(2)},s)$  are continuous with probability 1. We may hence apply the Fundamental Theorem of Calculus and differentiate 33 to conclude that a.s.  $\tilde{\mathbf{h}} (\mathbf{x}_s^{(2)},s) = \mathbf{h}_1(\mathbf{x}_s^{(2)},s)$  for all  $s$ .
As  $g(0) > 0$ ,  $\mathbf{x}_t^{(2)}$  is supported on all of  $\mathbb{R}^n$  for any  $t > 0$ . Therefore, the above implies that  $\tilde{\mathbf{h}} (\mathbf{x},s) = \mathbf{h}_1(\mathbf{x},s)$  for all  $s\in (0,T]$  and all  $\mathbf{x}\in \mathbb{R}^n$ .
More formally, let  $\Delta(\mathbf{x}', s) = \tilde{\mathbf{h}}(\mathbf{x}', s) - \mathbf{h}_1(\mathbf{x}', s)$ . Then  $\Delta(\mathbf{x}_s^{(2)}, s) = 0$  for all  $s$  a.s.. Also,  $\Delta$  is Lipschitz. Let  $C$  be its Lipschitz constant. Take  $\mathbf{x} \in \mathbb{R}^n$  and fix  $\varepsilon > 0$  and  $t > 0$ . Then  $\mathbb{P}(|\mathbf{x}_t^{(2)} - \mathbf{x}|_2 < \frac{\varepsilon}{2C}) > 0$ , as  $\mathbf{x}_t^{(2)}$  is supported in all of  $\mathbb{R}^n$ . Hence, by the above, there exists  $\mathbf{y}_{\varepsilon} \in B(\mathbf{x}, \frac{\varepsilon}{2C})$  such that  $\Delta(\mathbf{y}_{\varepsilon}, t) = 0$ . As  $\tilde{\mathbf{h}}$  is Lipschitz, for any  $\mathbf{x}' \in B(\mathbf{x}, \frac{\varepsilon}{2C})$ , we have that
$$
\begin{array}{l} \left| \left| \Delta \left(\mathbf {x} ^ {\prime}, t\right) \right| \right| \leq C \left| \left| \mathbf {x} ^ {\prime} - \mathbf {y} _ {\varepsilon} \right| \right| (34) \\ \leq C \left(\left| | \mathbf {x} ^ {\prime} - \mathbf {x} | | + | | \mathbf {x} - \mathbf {y} _ {\varepsilon} | |\right) \right. (35) \\ \leq C \left(\frac {\varepsilon}{2 C} + \frac {\varepsilon}{2 C}\right) (36) \\ = \varepsilon (37) \\ \end{array}
$$
As  $\varepsilon$  was arbitrary, we have that  $\Delta (\mathbf{x}^{\prime},t)\to 0$  as  $\mathbf{x}'\rightarrow \mathbf{x}$ . As  $\Delta$  is continuous (since it is Lipschitz), it follows that  $\Delta (\mathbf{x},t) = 0$ . As  $\mathbf{x}$  was also arbitrary, we have that  $\tilde{\mathbf{h}} (\mathbf{x},s) = \mathbf{h}_1(\mathbf{x},s)$  for all  $s\in (0,T]$  and all  $\mathbf{x}\in \mathbb{R}^n$ .
As  $\mathbf{h}$  must be chosen to be (Lipschitz) continuous also with respect to  $t$ , for any  $\mathbf{x}$  it must be that
$$
\tilde {\mathbf {h}} (\mathbf {x}, 0) = \lim  _ {s \rightarrow 0 ^ {+}} \tilde {\mathbf {h}} (\mathbf {x}, s) = \lim  _ {s \rightarrow 0 ^ {+}} \mathbf {h} (\mathbf {x}, s) = \mathbf {h} _ {1} (\mathbf {x}, 0) \tag {38}
$$
This completes the proof of the uniqueness of the choice  $\mathbf{h} = \mathbf{h}_1$ .
C.3 Does the Backward Process of a Diffusion Model Fit in Theorem C.1?
In the forward process 2,  $\mathbf{f}$  is taken to be Lipschitz, and  $g$  is always bounded, e.g. in [21, 62, 65]. However, in the backward process, one may be concerned about whether the score term  $-\nabla_{\mathbf{x}}\log p_t(\mathbf{x}_t,t)$  is Lipschitz. We argue that it is reasonable to assume that it is.
We consider a setting where  $g(t) = 1$ , and  $\mathbf{f}(\mathbf{x}, t) = -\lambda \mathbf{x}$ . Then the forward process is the well-understood Ornstein-Uhlenbeck (or Langevin) process [45, p. 74]:
$$
d \mathbf {x} _ {t} = - \lambda \mathbf {x} _ {t} d t + d \mathbf {w} _ {t} \tag {39}
$$
Consider the simplified case where the initial condition is deterministic:  $\mathbf{x}_0 = \mathbf{y} \in \mathbb{R}^n$ . Then  $\mathbf{x}$  is a Gaussian process of co-variance function given by  $K(s, t) = \operatorname{cov}(X_s, X_t) = \frac{\mathrm{e}^{-\lambda|t - s|} - \mathrm{e}^{-\lambda(t + s)}}{2\lambda} I$ , where  $I$  is the identity matrix, as per Le Gall [36], page 226. Its mean function is  $\mathbf{m}_t = \mathbf{y} e^{-\lambda t}$ . In particular, it has variance  $\sigma_t^2 = \frac{1 - \mathrm{e}^{-2\lambda t}}{2\lambda}$ , and  $\sigma_t^2 \to \frac{1}{2\lambda}$  as  $t \to \infty$ .
We follow the theoretical analysis of the convergence of diffusion models by De Bortoli [9] and assume the backward diffusion process finishes at a time  $\varepsilon > 0$ . Hence, the backward process  $\bar{\mathbf{x}}_t$  is now supported in all of  $\mathbb{R}^n$  for any  $t \in [\epsilon, T]$ . The original data need not be supported everywhere, and in fact is often assumed to not be, c.f. the manifold hypothesis in [9]. In our simplified case, it is supported at a point. This is why  $t = 0$  needs to be excluded by the backward process finishing at some  $\varepsilon > 0$ .
From the aforementioned properties of the Ornstein-Uhlenbeck process,  $p_t(\mathbf{x}_t) = C \exp \left( -\frac{||\mathbf{x}_t - \mathbf{m}_t||_2^2}{2\sigma_t^2} \right)$ , where  $C$  is a constant independent of  $\mathbf{x}$ . But then  $-\nabla_{\mathbf{x}} \log p_t(\mathbf{x}_t, t) = \frac{\mathbf{x}_t - \mathbf{m}_t}{\sigma_t^2}$ . For any fixed  $t$ , we hence have that  $-\nabla_{\mathbf{x}} \log p_t(\mathbf{x}_t, t)$  is  $1 / \sigma_t^2$ -Lipschitz. Since the variance  $\sigma_t$  is bounded away from 0 for  $t \in [\varepsilon, T]$ , we may pick a Lipschitz constant that works uniformly across all  $\mathbf{x}$  and  $t$  (for instance,  $1 / \sigma_{\varepsilon}^2$ ).
This can be extended to the case where the initial data is bounded, which is frequently assumed (e.g. by De Bortoli [9]). Informally, if the data is bounded, it cannot change the tails of the distribution of  $\mathbf{x}_t$  significantly. We omit further details, as this discussion is beyond the scope of the paper.
In summary, the forward noising process is very similar to an Ornstein-Uhlenbeck process, so, for practical purposes, it is reasonable to assume the scores  $\nabla_{\mathbf{x}}\log p_t(\mathbf{x}_t,t)$  are Lipschitz.
C.4 Existence and Uniqueness of Minimizer of (11)
Definition C.4.  $L^2 (\mathbb{R}^n,\mathbb{R}^n)$  is the Hilbert space given by
$$
L ^ {2} \left(\mathbb {R} ^ {n}, \mathbb {R} ^ {n}\right) = \left\{\mathbf {f}: \mathbb {R} ^ {n} \rightarrow \mathbb {R} ^ {n}: \int_ {\mathbb {R} ^ {n}} | | \mathbf {f} (\mathbf {x}) | | _ {2} ^ {2} d \mathbf {x} <   \infty \right\} \tag {40}
$$
Remark C.5. It is easy to show  $L^2(\mathbb{R}^n, \mathbb{R}^n)$  is a Hilbert Space with norm given by
$$
\left| \left| \mathbf {f} \right| \right| _ {2} ^ {2} = \int_ {\mathbb {R} ^ {n}} \left| \left| \mathbf {f} (\mathbf {x}) \right| \right| _ {2} ^ {2} d \mathbf {x} \tag {41}
$$
For instance, one can start from the standard result that  $L^2(\mathbb{R}^n, \mathbb{R})$  is a Hilbert Space, and apply it to each coordinate of the output of  $\mathbf{f}$ .
Definition C.6. Denote by  $\mathrm{Cons}(\mathbb{R}^n)$  the space of the gradients of smooth, square-integrable potentials from  $\mathbb{R}^n$  to  $\mathbb{R}$  with square-integrable gradient, i.e.
$$
\operatorname {C o n s} \left(\mathbb {R} ^ {n}\right) = \left\{\nabla f: f \text {i s s m o o t h ,} \int_ {\mathbb {R} ^ {n}} | f (\mathbf {x}) | ^ {2} d \mathbf {x} <   \infty \text {a n d} \int_ {\mathbb {R} ^ {n}} | | \nabla f (\mathbf {x}) | | _ {2} ^ {2} d \mathbf {x} <   \infty \right\} \tag {42}
$$
Remark C.7. Clearly  $\operatorname{Cons}(\mathbb{R}^n) \subseteq L^2(\mathbb{R}^n, \mathbb{R}^n)$ . The condition that " $f$  is square-integrable and has a square-integrable gradient" corresponds to  $f$  being in the Sobolev space  $W^{1,2}(\mathbb{R}^n)$  (see Jost [29], Chapter 9).
Denote by  $\overline{\operatorname{Cons}(\mathbb{R}^n)}$  the closure of  $\operatorname{Cons}(\mathbb{R}^n)$ , i.e.
$$
\overline {{\operatorname {C o n s} \left(\mathbb {R} ^ {n}\right)}} = \left\{\mathbf {f}: \text {t h e r e e x i s t s} (\nabla f _ {k}) _ {k \geq 0} \subseteq \operatorname {C o n s} \left(\mathbb {R} ^ {n}\right) \text {s u c h t h a t} | | \nabla f _ {k} - \mathbf {f} | | _ {2} ^ {2} \rightarrow 0 \text {a s} k \rightarrow \infty \right\} \tag {43}
$$
By construction,  $\overline{\mathrm{Cons}} (\mathbb{R}^n)$  is a vector subspace of  $L^2 (\mathbb{R}^n,\mathbb{R}^n)$ , and is closed, i.e. stable under taking limits.
Theorem C.8 (Complementation in Hilbert Spaces, c.f. Kreyszig [34], Theorem 3.3-4). Let  $(\mathcal{H},\langle \cdot ,\cdot \rangle ,||\cdot ||)$  be a Hilbert space, and let  $Y\subseteq H$  be a closed vector subspace. Then any  $x\in \mathcal{H}$  can be written as  $x = y + z$  where  $y\in Y$  and  $z\in Y^{\perp}$
Corollary C.9 (Uniqueness of Projection). Then the minimum of  $v \mapsto ||v - x||$  over  $v \in Y$  is attained at the (unique)  $y$  given in the theorem, as  $||v - x||^2 \geq |\langle v - x, z \rangle| = ||z||^2$ , and setting  $v = y$  attains this bound.
Proof of Proposition 4.3. Firstly note that we assumed  $\mathbf{s}_{\Theta}^{(2)}(\cdot ,t) - \mathbf{s}_{\phi}^{(1)}(\cdot ,t)$  is in  $L^2 (\mathbb{R}^n,\mathbb{R}^n)$  for each individual  $t$
It follows directly from the above that, as  $\overline{\operatorname{Cons}(\mathbb{R}^n)}$  is a closed subspace of  $L^2 (\mathbb{R}^n,\mathbb{R}^n)$ , Corollary C.9 applies, and we have that there is a unique minimizer  $\mathbf{h}_t\in \overline{\operatorname{Cons}(\mathbb{R}^n)}$  in Equation 11.
As  $\overline{\mathrm{Cons}} (\mathbb{R}^n)$  consists of  $L^2 (\mathbb{R}^n,\mathbb{R}^n)$  limits of sequences in  $\mathrm{Cons}(\mathbb{R}^n)$ , there exists a sequence  $(\nabla \Phi_k)_{k\geq 0}$  of gradients of  $W^{1,2}$ -potentials such that  $||\nabla \Phi_k - \mathbf{h}_t||_2^2\to 0$  as  $k\to \infty$ . From the definition of convergence, we
get that, for any  $\varepsilon > 0$ , there is  $k$  large enough such that
$$
\int_ {\mathbb {R} ^ {n}} \left\| \nabla_ {\mathbf {x}} \Phi_ {k} (\mathbf {x}) - \mathbf {h} _ {t} (\mathbf {x}) \right\| _ {2} ^ {2} d \mathbf {x} <   \varepsilon \tag {44}
$$
Appendix D Relative Reward Learning Algorithms
The below algorithm was used for learning the relative reward functions for experiments with Stable Diffusion (see Section 5.3).