SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Giulio Franzese  
EURECOM, France
Giulio Corallo  
EURECOM, France
Simone Rossi*  
Stellantis, France
Markus Heinonen  
Aalto University, Finland
Maurizio Filippone  
EURECOM, France
Pietro Michiardi  
EURECOM, France
We introduce Functional Diffusion Processes (FDPs), which generalize score-based diffusion models to infinite-dimensional function spaces. FDPs require a new mathematical framework to describe the forward and backward dynamics, and several extensions to derive practical training objectives. These include infinite-dimensional versions of Girsanov theorem, in order to be able to compute an ELBO, and of the sampling theorem, in order to guarantee that functional evaluations in a countable set of points are equivalent to infinite-dimensional functions. We use FDPs to build a new breed of generative models in function spaces, which do not require specialized network architectures, and that can work with any kind of continuous data. Our results on real data show that FDPs achieve high-quality image generation, using a simple MLP architecture with orders of magnitude fewer parameters than existing diffusion models. Code available here.
Diffusion models have recently gained a lot of attention both from academia and industry. The seminal work on denoising diffusion (Sohl-Dickstein et al., 2015) has spurred interest in the understanding of such models from several perspectives, ranging from denoising autoencoders (Vincent, 2011) with multiple noise levels (Ho et al., 2020), variational interpretations (Kingma et al., 2021), annealed (Song & Ermon, 2019) and continuous-time score matching (Song & Ermon, 2020; Song et al., 2021). Several recent extensions of the theory underpinning diffusion models tackle alternatives to Gaussian noise (Bansal et al., 2022; Rissanen et al., 2022), second order dynamics (Dockhorn et al., 2022), and improved training and sampling (Xiao et al., 2022; Kim et al., 2022b; Franzese et al., 2022).
Diffusion models have rapidly become the go-to approach for generative modeling, surpassing GANs (Dhariwal & Nichol, 2021) for image generation, and have recently been applied to various modalities such as audio (Kong et al., 2021; Liu et al., 2022), video (Ho et al., 2022; He et al., 2022), molecular structures and general 3D shapes (Tripe et al., 2022; Hoogeboom et al., 2022; Luo & Hu, 2021; Zeng et al., 2022). Recently, the generation of diverse and realistic data modalities (images, videos, sound) from open-ended text prompts (Ramesh et al., 2022; Sahara et al., 2022; Rombach et al., 2022) has projected practitioners into a whole new paradigm for content creation.
A common trait of diffusion models is the need to understand their design space (Karras et al., 2022), and tailor the inner working parts to the chosen application and data domain. Diffusion models require specialization, ranging from architectural choices of neural networks used to approximate the score (Dhariwal & Nichol, 2021; Karras et al., 2022), to fine-grained details such as an appropriate definition of a noise schedule (Dhariwal & Nichol, 2021; Salimans & Ho, 2022), and mechanisms to deal with resolution and scale (Ho et al., 2021). Clearly, the data domain impacts profoundly such design choices. As a consequence, a growing body of work has focused on the projection of data
*This work was done while working at EURECOM
modalities into a latent space (Rombach et al., 2022), either by using auxiliary models such as a VAEs (Vahdat et al., 2021), or by using a functional data representation (Dupont et al., 2022a). These approaches lead to increased efficiency, because they operate on smaller dimensional spaces, and constitute a step toward broadening the applicability of diffusion models to general data.
The idea of modelling data with continuous functions has several advantages (Dupont et al., 2022a): it allows working with data at arbitrary resolutions, it enjoys improved memory-efficiency, and it allows simple architectures to represent a variety of data modalities. However, a theoretically grounded understanding of how diffusion models can operate directly on continuous functions has been elusive so far. Preliminary studies apply established diffusion algorithms on a discretization of functional data by conditioning on point-wise values (Dutordoir et al., 2022; Zhuang et al., 2023). A line of work that is closely related to ours include approaches such as Kerrigan et al. (2022), who consider a Gaussian noise corruption process in Hilbert space and derive a loss function formulated on infinite-dimensional measures to approximate the conditional mean of the reverse process. Within this line of works, Mittal et al. (2022) consider diffusion of Gaussian processes. We are aware of other concurrent works that study diffusion process in Hilbert spaces (Lim et al., 2023; Pidstrigach et al., 2023; Hagemann et al., 2023). However, differently from us, these works do not formally prove that the score matching optimization is a proper evidence lower bound (ELBO), but simply propose it as an heuristic. None of these prior works discuss the limits of discretization, resulting in the failure of identifying which subset of functions can be reconstructed through sampling. Finally, the parametrization we present in our work merges how functions and score are approximated using a single, simple model.
The main goal of our work is to deepen our understanding of diffusion models in function space. We present a new mathematical framework to lift diffusion models from finite-dimensional inputs to function spaces, contributing to a general method for data represented by continuous functions.
In § 2, we present Functional Diffusion Processes (FDPs), which generalize diffusion processes to infinite-dimensional functional spaces. We define forward (§ 2.1) and backward (§ 2.2) FDPs, and consider generic functional perturbations, including noising and Laplacian blurring. Using an extension of Girsanov theorem, we derive in § 2.3 an ELBO, which allows defining a parametric model to approximate the score of the functional density of FDPs. Given a FDP and the associated ELBO, we are one-step closer to the definition of a loss function to learn the parametric score. However, our formulation still resides in an abstract, infinite-dimensional Hilbert space.
Then, for practical reasons, in § 3, we specify for which subclass of functions we can perfectly reconstruct the original function given only its evaluation in a countable set of points. This is an extension of the sampling theorem, which we use to move from the infinite-dimensional domain of functions to a finite-dimensional domain of discrete mappings.
In § 4, we discuss various options to implement such discrete mappings. In this work, we explore in particular the usage of implicit neural representations (INRs) (Sitzmann et al., 2020) and Transformers Vaswani et al. (2017) to jointly model both the sampled version of infinite-dimensional functions, and the score network, which is central to the training of FDPs, and is required to simulate the backward process. Our training procedure, discussed in § 5, involves approximate, finite-dimensional Stochastic Differential Equations (SDEs) for the forward and backward processes, as well as for the ELBO.
We complement our theory with a series of experiments to illustrate the viability of FDPs, in § 6. In our experiments, the score network is a simple multilayer perceptron (MLP), with several orders of magnitude fewer parameters than any existing score-based diffusion model. To the best of our knowledge, we are the first to show that a functional-space diffusion model can generate realistic image data, beyond simple data-sets and toy models.
2 Functional Diffusion Processes (FDPs)
We begin by defining diffusion processes in Hilbert Spaces, which we call functional diffusion processes (FDPs). While the study of diffusion processes in Hilbert spaces is not new (Follmer & Wakolbinger, 1986; Millet et al., 1989; Da Prato & Zabczyk, 2014), our objectives depart from prior work, and call for an appropriate treatment of the intricacies of FDPs, when used in the context of generative modeling. In § 2.1 we introduce a generic class of diffusion processes in Hilbert spaces. The key object is Equation (1), together with its associated path measure  $\mathbb{Q}$  and the time varying
measure  $\rho_{t}$ , where  $\rho_0$  represents the starting (data) measure. In § 2.2 we derive the reverse FDP with the associated path-reversed measure  $\hat{\mathbb{Q}}$ , and in § 2.3 we use an extension of Girsanov theorem for infinite-dimensional SDEs to compute the ELBO. The ELBO is a training objective involving a generalization of the score function (Song et al., 2021).
2.1 The forward diffusion process
We consider  $H$  to be a real, separable Hilbert space with inner product  $\langle \cdot ,\cdot \rangle$ , norm  $\| \cdot \| _H$ , and countable orthonormal basis  $\{e^k\}_{k = 1}^\infty$ . Let  $L(H)$  be the set of bounded linear operators on  $H$ ,  $B(H)$  be its Borel  $\sigma$ -algebra,  $B_{b}(H)$  be the set of bounded  $B(H)$ -measurable functions  $H\to \mathbb{R}$ , and  $P(H)$  be the set of probability measures on  $(H,B(H))$ . Consider the following  $H$ -valued SDE:
$$
\left\{ \begin{array}{l} \mathrm {d} X _ {t} = \left(\mathcal {A} X _ {t} + f \left(X _ {t}, t\right)\right) \mathrm {d} t + \mathrm {d} W _ {t}, \\ X _ {0} \sim \rho_ {0} \in P (H), \end{array} \right. \tag {1}
$$
where  $t \in [0, T]$ ,  $W_t$  is a  $R$ -Wiener process on  $H$  defined on the quadruplet  $(\Omega, \mathcal{F}, (\mathcal{F}_t)_{t \geq 0}, \mathbb{Q})$ , and  $\Omega, \mathcal{F}$  are the sample space and canonical filtration, respectively. The domain of  $f$  is  $D(f) \in B([0, T] \times H)$ , where  $f: D(f) \subset [0, T] \times H \to H$  is a measurable map. The operator  $\mathcal{A}: D(\mathcal{A}) \subset H \to H$  is the infinitesimal generator of a  $C_0$ -semigroup  $\exp(t\mathcal{A})$  in  $H$  ( $t \geq 0$ ), and  $\rho_0$  is a probability measure in  $H$ . We consider  $\Omega$  to be  $C^1([0, T])$ , that is the space of all continuous mappings  $[0, T] \to H$ , and  $X_t(\omega) = \omega(t)$ ,  $\omega \in \Omega$  to be the canonical process. The requirements on the terms  $\mathcal{A}, f$  that ensure existence of solutions to Equation (1) depend on the type of noise — trace-class  $(\mathrm{Tr}\{R\} < \infty)$  or cylindrical ( $R = I$ ) — used in the FDP (Da Prato & Zabczyk (2014), Hypothesis 7.1 or Hypothesis 7.2 for trace-class and cylindrical noise, respectively).
The measure associated with Equation (1) is indicated with  $\mathbb{Q}$ . The law induced at time  $\tau \in [0,T]$  by the canonical process on the measure  $\mathbb{Q}$  is indicated with  $\rho_{\tau} \in P(H)$ , where  $\rho_{\tau}(S) = \mathbb{Q}(\{\omega \in \Omega : X_{\tau}(\omega) \in S\})$ , and  $S$  is any element of  $\mathcal{F}$ . Notice, that in infinite dimensional spaces there is not an equivalent of the Lebesgue measure to get densities from measures. In our case we consider however, when it exists, the single-dimensional density  $\rho_{\tau}^{(d)}(x^i | x^{j \neq i})$ , defined implicitly through  $\mathrm{d}\rho_{\tau}(x^i | x^{j \neq i}) = \rho_{\tau}^{(d)}(x^i | x^{j \neq i})\mathrm{d}x^i$ , being  $\mathrm{d}x^i$  the Lebesgue measure. To avoid cluttering the notation, in this work we simply shorten  $\rho_{\tau}^{(d)}(x^i | x^{j \neq i})$  with  $\rho_{\tau}^{(d)}(x)$  whenever unambiguous. In Appendix B we provide additional details on the time-varying measure  $\rho_t(\mathrm{d}x)\mathrm{d}t$ . Before proceeding, it is useful to notice that Equation (1) can also be expressed as an (infinite) system of stochastic differential equations in terms of  $X_{t}^{k} = \langle X_{t}, e^{k} \rangle$  as:
$$
\mathrm {d} X _ {t} ^ {k} = b ^ {k} \left(X _ {t}, t\right) \mathrm {d} t + \mathrm {d} W _ {t} ^ {k}, \quad k = 1, \dots , \infty , \tag {2}
$$
where we introduced the projection  $b^{k}(X_{t},t) = \langle \mathcal{A}X_{t} + f(X_{t},t),e^{k}\rangle$ . Moreover,  $\mathrm{d}W_t^k = \langle dW_t,e^k\rangle$  with covariance given by  $\mathbb{E}[W_t^k W_s^j ] = \delta (k - j)r^k\min (s,t)$ ,  $\delta$  in Kroenecker sense, and  $r^k$  is the projection on the base of  $R$ .
2.2 The reverse diffusion process
We now derive the reverse time dynamics for FDPs of the form defined in Equation (1). We require that the time reversal of the canonical process,  $\hat{X}_t = X_{T - t}$ , is again a diffusion process, with distribution given by the path-reversed measure  $\hat{\mathbb{Q}} (\omega)$ , along with the reversed filtration  $\hat{\mathcal{F}}$ . Note that the time reversal of an infinite dimensional process is more involved than for the finite dimensional case (Anderson, 1982; Föllmer, 1985). There are two major approaches to guarantee the existence of the reverse diffusion process. The first approach (Föllmer & Wakolbinger, 1986) is applicable only when  $R = I$  (the case of cylindrical Wiener processes) and it relies on a finite local entropy condition. The second approach, which is valid in the case of trace class noise  $\mathrm{Tr}\{R\} < \infty$ , is based on stochastic calculus of variations (Millet et al., 1989). The full technical analysis of the necessary assumptions for the two approaches is involved, and we postpone formal details to Appendix A.
Theorem 1. Consider Equation (1). If  $R = I$ , suppose Assumption 1 in Appendix A.1 holds; else,  $(R \neq I)$  suppose Assumption 5 and Assumption 6 in Appendix A.2 hold. Then  $\hat{X}_t$ , corresponding to the path measure  $\hat{\mathbb{Q}}(\omega)$ , has the following SDE representation:
$$
\left\{ \begin{array}{l} \mathrm {d} \hat {X} _ {t} = \left(- \mathcal {A} \hat {X} _ {t} - f (\hat {X} _ {t}, T - t) + R D _ {x} \log \rho_ {T - t} ^ {(d)} (\hat {X} _ {t})\right) \mathrm {d} t + \mathrm {d} \hat {W} _ {t}, \\ \hat {X} _ {0} \sim \rho_ {T}, \end{array} \right. \tag {3}
$$
where  $\hat{W}$  is a  $\hat{\mathbb{Q}}$ $R$ -Wiener process, and the notation  $D_{x}\log \rho_{t}^{(d)}(x)$  stands for the mapping  $H\to H$  that, when projected, satisfies  $\langle D_x\log \rho_t^{(d)}(x),e^k\rangle = \frac{\partial}{\partial x^k}\log \Bigl (\rho_t^{(d)}(x^k\mid x^{i\neq k})\Bigr)$ .
By projecting onto the eigenbasis, we have an infinite system of SDEs:
$$
\mathrm {d} \hat {X} _ {t} ^ {k} = \left(- b ^ {k} (\hat {X} _ {t}, T - t) + r ^ {k} \frac {\partial}{\partial x ^ {k}} \log \left(\rho_ {T - t} ^ {(d)} (\hat {X} _ {t})\right)\right) \mathrm {d} t + \mathrm {d} \hat {W} _ {t} ^ {k}, k = 1, \dots , \infty . \tag {4}
$$
The methodology proposed in this work requires to operate on proper Wiener processes, with  $\mathrm{Tr}\{R\} < \infty$ , which implies, intuitively, that the considered noise has finite variance. We now discuss a Corollary, in which Assumption 5 is replaced by stricter conditions, that we use to check the validity of the practical implementation of FDPs.
Corollary 1. Suppose Assumption 6 from Appendix A.2 holds. Assume that  $i)$ $\mathrm{Tr}\{R\} = \sum_{i}r^{i} < \infty$  ii)  $b^{i}(x,t) = b^{i}x^{i},\forall i$  , i.e. the drift term is linear and only depends on  $x$  through its projection onto the corresponding basis and iii) the drift is bounded, such that  $\exists K > 0: - K <   b^i <  0,\forall i$  . Then, the reverse process evolves according to Equation (4).
Theorem 1 stipulates that, given some conditions, the reverse time dynamics for FDPs of the form defined in Equation (1) exist. Our analysis provides theoretical grounding to the observations in concurrent work (Lim et al., 2023) where, empirically, it is observed that the cylindrical class of noise is not suitable. We argue that, when  $R = I$ , the difficulty stems from designing the coefficients  $b^{i}$  of the SDEs such that the forward (see requirement (5.3) in Da Prato & Zabczyk (2014)) as well as the backward processes Assumption 1 exist. The work by Bond-Taylor & Willcocks (2023) uses cylindrical (white) noise, but we are not aware of any theoretical justification, since the model architecture is only partially suited for the functional domain.
As an addendum, we note that the advantages of projecting the forward and backward processes on the eigenbasis of the Hilbert space  $H$ , as in Equation (2) and Equation (4), become evident when discussing about the implementation of FDPs, specifically when we derive practical expressions for training and the simulation of the backward process, as discussed in § 5, and in a fully expanded toy example in Appendix D.
2.3 A Girsanov formula for the ELBO
Direct simulation of the backward FDP described by Equation (3) is not possible. Indeed, we have no access to the true score of the density  $\rho_{\tau}^{(d)}$  induced at time  $\tau \in [0,T]$ . To solve the problem, we introduce a parametric score function  $s_\theta : H \times [0,T] \times \mathbb{R}^m \to H$ . We consider the dynamics:
$$
\left\{ \begin{array}{l} \mathrm {d} \hat {X} _ {t} = \left(- \mathcal {A} \hat {X} _ {t} - f (\hat {X} _ {t}, T - t) + R s _ {\boldsymbol {\theta}} (\hat {X} _ {t}, T - t)\right) \mathrm {d} t + \mathrm {d} \tilde {W} _ {t}, \\ \hat {X} _ {0} \sim \chi_ {T} \in P (H), \end{array} \right. \tag {5}
$$
with path measure  $\hat{\mathbb{P}}^{\chi_T}$ , and  $\mathrm{d}\tilde{W}_t$  being a  $\hat{\mathbb{P}}^{\chi_T}R$ -Wiener process. To emphasize the connection between Equation (3) and Equation (5), we define initial conditions with the subscript  $T$ , instead of 0. In principle, we should have  $\chi_T = \rho_T$ , as it will be evident from the ELBO in Equation (8). However,  $\rho_T$  has a simple and easy-to-sample-from distribution only for  $T\to \infty$ , which is not compatible with a realistic implementation. The analysis of the discrepancy when  $T$  is finite is outside of the scope of this work, and the interested reader can refer to Franzese et al. (2022) for an analysis on standard diffusion models. The final measure of the new process at time  $T$  is indicated by  $\chi_0$ , i.e.  $\chi_0(S) = \hat{\mathbb{P}}^{\chi_T}(\{\omega \in \Omega : \hat{X}_T(\omega) \in S\})$ .
Next, we quantify the discrepancy between  $\chi_0$  and the true data measure  $\rho_0$  through an ELBO. Thanks to an extension of Girsanov theorem to infinite dimensional SDEs (Da Prato & Zabczyk, 2014), it is possible to relate the path measures ( $\hat{\mathbb{Q}}$  and  $\hat{\mathbb{P}}^{\chi_T}$ , respectively) of the process  $\hat{X}_t$  induced by different drift terms in Equation (3) and different initial conditions.
Starting from the score function  $s_{\theta}$ , we define:
$$
\gamma_ {\boldsymbol {\theta}} (x, t) = R \left(s _ {\boldsymbol {\theta}} (x, T - t) - D _ {x} \log \rho_ {T - t} ^ {(d)} (x)\right). \tag {6}
$$
Under loose regularity assumptions (see Condition 2 in Appendix A.4)  $\tilde{W}_t = \hat{W}_t - \intop_0^t\gamma_\theta (X_s,s)ds$  is a  $\hat{\mathbb{P}}^{\rho_T}R$ -Wiener process (Theorem 10.14 in Da Prato & Zabczyk (2014)), where Girsanov Theorem also tells us that the measure  $\hat{\mathbb{P}}^{\rho_T}$  satisfies the Radon-Nikodym derivative:
$$
\frac {\mathrm {d} \hat {\mathbb {P}} ^ {\rho_ {T}}}{\mathrm {d} \hat {\mathbb {Q}}} = \exp \left(\int_ {0} ^ {T} \left\langle \gamma_ {\boldsymbol {\theta}} \left(\hat {X} _ {t}, t\right), \mathrm {d} \hat {W} _ {t} \right\rangle_ {R ^ {\frac {1}{2}} H} - \frac {1}{2} \int_ {0} ^ {T} \left\| \gamma_ {\boldsymbol {\theta}} \left(\hat {X} _ {t}, t\right) \right\| _ {R ^ {\frac {1}{2}} H} ^ {2} \mathrm {d} t\right). \tag {7}
$$
By virtue of the disintegration theorem,  $\mathrm{d}\hat{\mathbb{Q}} = \mathrm{d}\hat{\mathbb{Q}}_0\mathrm{d}\rho_T$  and similarly  $\mathrm{d}\hat{\mathbb{P}}^{\rho_T} = \mathrm{d}\hat{\mathbb{P}}_0\mathrm{d}\rho_T$ , being  $\hat{\mathbb{Q}}_0,\hat{\mathbb{P}}_0$  the measures of the processes when considering a particular initial value. Then,  $\hat{\mathbb{P}}^{\chi_T}$  satisfies  $\mathrm{d}\hat{\mathbb{P}}^{\chi_T} = \mathrm{d}\hat{\mathbb{P}}\frac{\mathrm{d}\chi_T}{\mathrm{d}\rho_T}$ , for any measure  $\chi_T\in P(H)$ . Consequently, the canonical process  $\hat{X}_t$  has an SDE representation according to Equation (5), under the new path measure  $\hat{\mathbb{P}}^{\chi_T}$ . Then (see Appendix A.5 for the derivation) we obtain the ELBO:
$$
\mathrm {K L} \left[ \rho_ {0} \| \chi_ {0} \right] \leq \frac {1}{2} \mathbb {E} _ {\mathbb {Q}} \left[ \int_ {0} ^ {T} \| \gamma_ {\boldsymbol {\theta}} \left(X _ {t}, t\right) \| _ {R ^ {\frac {1}{2}} H} ^ {2} \mathrm {d} t \right] + \mathrm {K L} \left[ \rho_ {T} \| \chi_ {T} \right]. \tag {8}
$$
Provided that the required assumptions in Theorem 1 are met, the validity of Equation (8) is general. Our goal, however, is to set the stage for a practical implementation of FDPs, which calls for design choices that easily enable satisfying the required assumptions for the theory to hold. Then, for the remainder of the paper, we consider the particular case where  $f = 0$  in Equation (1). This simplifies the dynamics as follows:
$$
\mathrm {d} X _ {t} = \mathcal {A} X _ {t} \mathrm {d} t + \mathrm {d} W _ {t}, \quad X _ {0} \sim \rho_ {0} \in P (H) \tag {9}
$$
$$
\mathrm {d} \hat {X} _ {t} = \left(- \mathcal {A} \hat {X} _ {t} + R s _ {\boldsymbol {\theta}} (\hat {X} _ {t}, T - t)\right) \mathrm {d} t + \mathrm {d} \tilde {W} _ {t}, \quad \hat {X} _ {0} \sim \chi_ {T} \in P (H) \tag {10}
$$
Since the only drift component in Equation (9) is the linear term  $\mathcal{A}$ , the projection  $b^{j}$  will be linear as well. Such a design choice, although not necessary from a theoretical point of view, carries several advantages. The design of a drift term satisfying the conditions of Corollary 1 becomes straightforward, where such conditions naturally aligns with the requirements of the existence of the forward process (Chapter 5 of Da Prato & Zabczyk (2014)). Moreover, the forward process conditioned on given initial conditions admits known solutions, which means that simulation of SDE paths is cheap and straightforward, without the need for performing full numerical integration. Finally, it is possible to claim existence of the true score function and even provide its analytic expression (full derivation in Appendix A.7) as:
$$
D _ {x} \log \rho_ {t} ^ {(d)} (x) = - \mathcal {S} (t) ^ {- 1} \left(x - \exp (t \mathcal {A}) \mathbb {E} \left[ X _ {0} \mid X _ {t} = x \right]\right), \tag {11}
$$
where  $\mathcal{S}(t) = \left(\intop_{s = 0}^{t}\exp ((t - s)\mathcal{A})R\exp \bigl ((t - s)\mathcal{A}^{\dagger}\bigr)\mathrm{d}s\right)$ . This last aspect is particularly useful when considering the conditional version of Equation (6), through  $\langle D_x\log \rho_t^{(d)}(x|x_0),e^k\rangle = \frac{\partial}{\partial x^k}\log \Bigl (\rho_t^{(d)}(x^k\mid x^{i\neq k},x_0)\Bigr)$ , as:
$$
\tilde {\gamma} _ {\boldsymbol {\theta}} (x, x _ {0}, t) = R \left(s _ {\boldsymbol {\theta}} (x, T - t) - D _ {x} \log \rho_ {T - t} ^ {(d)} (x \mid x _ {0})\right), \tag {12}
$$
where, similarly to the unconditional case, we have  $D_x\log \rho_t^{(d)}(x\mid x_0) = -\mathcal{S}(t)^{-1}\left(x - \exp (t\mathcal{A})x_0\right)$ . Then, Equation (12) can be used to rewrite Equation (8):
$$
\mathbb {E} _ {\mathbb {Q}} \left[ \int_ {0} ^ {T} \| \gamma_ {\boldsymbol {\theta}} (X _ {t}, t) \| _ {R ^ {\frac {1}{2}} H} ^ {2} \mathrm {d} t \right] = \mathbb {E} _ {\mathbb {Q}} \left[ \int_ {0} ^ {T} \| \tilde {\gamma} _ {\boldsymbol {\theta}} (X _ {t}, X _ {0}, t) \| _ {R ^ {\frac {1}{2}} H} ^ {2} \mathrm {d} t \right] + I, \tag {13}
$$
where  $I$  is a quantity independent of  $\theta$ . Knowledge of the conditional true score  $D_{x}\log \rho_{t}^{(d)}(x\mid x_{0})$  and cheap simulation of the forward dynamics, allows for easier numerical optimization than the more general case of  $f\neq 0$ .
3 Sampling theorem for FDPs