SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For all our experiments, we rely on the official open-sourced code and the training and evaluation hyper-parameters from the paper "Elucidating the Design Space of Diffusion-Based Generative Models" [28] that, to the best of our knowledge, holds the current state-of-the-art on conditional generation on CIFAR-10 and unconditional generation on CIFAR-10, AFHQ  $(64 \times 64$  resolution), FFHQ  $(64 \times 64$  resolution). We refer to the models trained with our regularization as "CDM (Ours)" and to models trained with vanilla Denoising Score Matching (DSM) as "EDM" models. "CDM" models are trained with the weighted objective:
$$
L _ {\lambda} ^ {\mathrm {o u r s}} (\theta) = \mathbb {E} _ {t} \left[ \mathbb {E} _ {x _ {0} \sim p _ {0}, x _ {t} \sim \mathcal {N} (x _ {0}, \sigma_ {t} ^ {2} I _ {d})} L _ {t, x _ {t}, x _ {0}} ^ {\mathrm {S M}} (\theta) + \lambda \mathbb {E} _ {x _ {t} \sim p _ {t}} \mathbb {E} _ {t ^ {\prime} \sim \mathcal {U} [ t - \epsilon , t ]} L _ {t, t ^ {\prime}, x _ {t}} ^ {\mathrm {C P}} (\theta) \right],
$$
while the "EDM" models are trained only with the first term of the outer expectation. We also denote in the name whether the models have been trained with the Variance Preserving (VP) [51, 18] or the Variance Exploding [51, 50, 49], e.g. we write EDM-VP. Finally, for completeness, we also report scores from the models of Song et al. [51], following the practice of the EDM paper. We refer to the latter baselines as "NCSNv3" baselines.
We train diffusion models, with and without our regularization, for conditional generation on CIFAR-10 and unconditional generation on CIFAR-10 and AFHQ  $(64 \times 64$  resolution). For the re-trained models on CIFAR-10, we use exactly the same training hyperparameters as in Karras et al. [28] and we verify that our re-trained models match (within  $1\%$  ) the FID numbers mentioned in the paper. For AFHQ, we dropped the batch size from the suggested value of 512 to 256 to save on computational
resources, which increased the FID from 1.96 (reported value) to 2.29. All models were trained for  $200\mathrm{k}$  iterations, as in Karras et al. [28]. Finally, we retrain a baseline model on FFHQ for  $150\mathrm{k}$  iterations and we finetune it for  $5\mathrm{k}$  steps using our proposed objective.
Implementation Choices and Computational Requirements. As mentioned, when enforcing CP, we are free to choose  $t'$  anywhere in the interval  $[0, t]$ . When  $t, t'$  are far away, sampling  $x_{t}'$  from the distribution  $p_{t'}^{\theta}(x_{t}' | x_{t})$  requires many sampling steps (to reduce discretization errors). Since this needs to be done for every Gradient Descent update, the training time increases significantly. Instead, we notice that local consistency implies global consistency. Hence, we first fix the number of sampling steps to run in every training iteration and then we sample  $t'$  uniformly in the interval  $[t - \epsilon, t]$  for some specified  $\epsilon$ . For all our experiments, we fix the number of sampling steps to 6 which roughly increases the training time needed by 1.5x. We train all our models on a DGX server with 8 A100 GPUs with 80GBs of memory each.
5.1 Consistency Property Testing
We are now ready to present our results. The first thing that we check is whether regularizing for CP actually leads to models that are more consistent with their predictions, as the property implies. Specifically, we want to check that the model trained with  $L_{\lambda}^{\mathrm{ours}}$  achieves lower Consistency error, i.e. lower  $L_{t,t',x_t}^{\mathrm{CP}}$ . To check this, we do the following two tests: i) we fix  $t = 1$  and we show how  $L_{t,t',x_t}^{\mathrm{CP}}$  changes as  $t'$  changes in [0, 1], ii) we fix  $t' = 0$  and we show how the loss is changing as you change  $t$  in [0, 1]. Intuitively, the first test shows how the violation of CP splits across the sampling process and the second test shows how much you finally  $(t' = 0)$  violate the property if the violation started at time  $t$ . The results are shown in Figures 1a, 1b, respectively, for the models trained on AFHQ. We include additional results for CIFAR-10, FFHQ in Figures 4, 5, 6, 7 of the Appendix. As shown, indeed regularizing for the CP Loss drops the  $L_{t,t',x_t}^{\mathrm{CP}}$  as expected. See Section C.1 for additional details and discussion.
(a) Consistency Property Testing on AFHQ. The plot illustrates how the Consistency Loss,  $L_{t,t',x_t}^{\mathrm{CP}}$ , behaves for  $t' = 0$ , as  $t$  changes.
Figure 1: Consistency Property Testing on AFHQ.
(b) Consistency Property Testing on AFHQ. The plot illustrates how the Consistency Loss,  $L_{t,t',xt}^{\mathrm{CP}}$ , behaves for  $t = 0$ , as  $t'$  changes.
Performance. We evaluate the performance of our models. Following the methodology of Karras et al. [28], we generate 150k images from each model and we report the minimum FID computed on three sets of 50k images each. We keep checkpoints during training and we report FID for 30k, 70k, 100k, 150k, 180k and 200k iterations in Table 1. We also report the best FID found for each model, after evaluating checkpoints every 5k iterations (i.e. we evaluate 40 models spanning 200k steps of training). As shown in the Table, the proposed CP regularization yields improvements throughout the training. In the case of CIFAR-10 (conditional and unconditional) where the re-trained baseline was trained with exactly the same hyperparameters as the models in the EDM [28] paper,
(a) Uncurated images by our model trained on AFHQ. FID: 2.21, NFEs: 79.
Figure 2: Comparison of uncurated images generated by two different models.
(b) Uncurated images by our conditional CIFAR-10 model. FID: 1.77, NFEs: 35.
Model	30k	70k	100k	150k	180k	200k	Best
CDM-VP (Ours)	AFHQ	3.00	2.44	2.30	2.31	2.25	2.44	2.21
EDM-VP (retrained)	3.27	2.41	2.61	2.43	2.29	2.61	2.26
EDM-VP (reported)*5	1.96
EDM-VE (reported)*	2.16
NCSNv3-VP (reported)*	2.58
NCSNv3-VE (reported)*	18.52
CDM-VP (Ours)	CIFAR-10 (cond.)	2.44	1.94	1.88	1.88	1.80	1.82	1.77
EDM-VP (retrained)	2.50	1.99	1.94	1.85	1.86	1.90	1.82
EDM-VP (reported)	1.79
EDM-VE (reported)	1.79
NCSNv3-VP (reported)	2.48
NCSNv3-VE (reported)	3.11
CDM-VP (Ours)	CIFAR-10 (uncond.)	2.83	2.21	2.14	2.08	1.99	2.03	1.95
EDM-VP (retrained)	2.90	2.32	2.15	2.09	2.01	2.13	2.01
EDM-VP (reported)	1.97
EDM-VE (reported)	1.98
NCSNv3-VP (reported)	3.01
NCSNv3-VE (reported)	3.77
CDM-VP (finetuned)	FFHQ	2.61
EDM-VP (retrained)	2.68
Table 1: FID results for deterministic sampling, using the Karras et al. [28] second-order samplers. For the CIFAR-10 models, we do 35 function evaluations and for AFHQ 79.
our CDM models achieve a new state-of-the-art. We further show that our CP regularization can be applied on top of a pre-trained model. Specifically, we train a baseline EDM-VP model on FFHQ  $64 \times 64$  for 150k using vanilla Denoising Score Matching. We then do 5k steps of finetuning, with and without our CP regularization and we measure the FID score of both models. The baseline model achieves FID 2.68 while the model finetuned with CP regularization achieves 2.61. This experiment shows the potential of applying our CP regularization to pre-trained models, potentially even at large scale, e.g. we could apply this idea with text-to-image models such as Stable Diffusion [42]. We leave this direction for future work.
Uncurated samples from our best models on AFHQ, CIFAR-10 and FFHQ are given in Figures 2a, 2b and 8. One benefit of the deterministic samplers is the unique identifiability property [51]. Intuitively, this means that by using the same noise and the same deterministic sampler, we can directly compare
visually models that might have been trained in completely different ways. We select a couple of images from Figure 2a (AFHQ generations) and we compare the generated images from our model with the ones from the EDM baseline for the same noises. The results are shown in Figure 3. As shown, the CP regularization fixes several geometric inconsistencies for the picked images. We underline that the shown images are examples for which CP regularization helped and that potentially there are images for which the baseline models give more realistic results.
We note that an appropriate regularization parameter  $\lambda$  for our consistency loss has to be chosen. We found that an excessively large  $\lambda$  harms the performance: while the CP regularization enforces the model to obey some diffusion process, it does not enforce it to obey the true diffusion process, and a large  $\lambda$  might disturb with the score-matching signal.
Figure 3: Visual comparison of EDM model (top) and CDM model (Ours, bottom) using deterministic sampling initiated with the same noise. As seen, the CP regularization fixes several geometric inconsistencies and artifacts in the generated images. In order to obtain this comparison, we generated 64 images both with EDM and CDM. We found that in 14 of these images, CDM provided considerable improvement and for the remainder there was no significant difference.
Model	FID
EDM (baseline)	5.81
CDM, all times t	5.45
CDM, for some t	6.59
CDM, for some t early stopped sampling	14.52
Table 2: Ablation study on removing the DSM loss for some  $t$  . Table reports FID results after  ${10}\mathrm{k}$  steps of training on CIFAR-10.
Ablation Study for Theoretical Predictions. One interesting implication of Theorem 3.2 is that it suggests that we only need to learn the score perfectly on some fixed  $t_0$  and then the CP implies that the score is learned everywhere (for all  $t$  and in the whole space). This motivates the following experiment: instead of using as our loss the weighted sum of DSM and our CP regularization for all  $t$ , we will not use DSM for  $t \leq t_{\mathrm{threshold}}$ , for some  $t_{\mathrm{threshold}}$  that we test our theory for.
We pick  $t_{\mathrm{threshold}}$  such that for 20% of the diffusion (on the side of clean images), we do not train with DSM. For the rest 80% we train with both DSM and our CP regularization. Since this is only an ablation study, we train for only 10k steps on (conditional) CIFAR-10. We report FID numbers for three models: i) training with only DSM, ii) training with DSM and CP regularization everywhere, iii) training with DSM for 80% of times  $t$  and CP regularization everywhere. In our reported models, we also include FID of an early stopped sampling of the latter model, i.e. we do not run the sampling for  $t < t_{\mathrm{threshold}}$  and we just output  $h_{\theta}(x_{t_{\mathrm{threshold}}}, t_{\mathrm{threshold}})$ . The numbers are summarized in Table 2. As shown, the theory is predictive since early stopping the generation at time  $t$  gives significantly worse results than continuing the sampling through the times that were never explicitly trained for approximating the score (i.e. we did not use DSM for those times). That said, the best results are obtained by combining DSM and our CP regularization everywhere, which is what we did for all the other experiments in the paper.
The fact that imperfect learning of the score function introduces a shift between the training and the sampling distribution has been well known. Chen et al. [6, 7] analyze how the  $l_{2}$  error in the approximation of the score function propagates to Total Variation distance error bounds between the true and the learned distribution. Several methods for mitigating this issue have been proposed, but the majority of the attempts focus on changing the sampling process [51, 28, 24, 46]. A related work is the Analog-Bits paper [8] that conditions the model during training with past model predictions.
Karras et al. [28] discusses potential violations of invariances, such as the non-conservativity of the induced vector field, due to imperfect score matching. However, they do not formally test or enforce this property. Chao et al. [5] shows that failure to satisfy the conservativity property can harm the performance, and they propose a modification to relieve this degradation. Lai et al. [33] study the problem of regularizing diffusion models to satisfy the Fokker-Planck equation. While we show in Theorem 3.2 that perfect conservative training enforces the Fokker-Planck equation, we notice that their training method is different: they suggest enforcing the equation locally by using the finite differences method to approximate the derivatives. Further, they do not train on drifted data. Instead, we notice that our CP loss is well suited to handle drifted data since it operates across trajectories generated by the model.
A concurrent work by Song et al. [52] proposes Consistency Models (CM), a new class of generative models that output directly the solution of the Probability Flow ODE. This idea resembles the consistency in the model outputs that we enforce through CP, but the motivation is different: CM attempts to accelerate sampling and we attempt to improve generation quality. The two methods are similar in that they enforce a property that the model should satisfy. In the case of the CM, the ODE solver should produce the same solution when evaluated at points that belong to the same trajectory. Our property is that on expectation the predictions should not be changing for points that have the same origin. One way to view it is that CM applies our consistency condition but on the deterministic sampler (for which the expectation becomes the point itself). Subsequent work by Lai et al. [34] compares our CDM to Consistency Models [52] and to Fokker Planck regularization [33].
We proposed an objective that enforces the trained network to follow a reverse Martingale, thereby having self-consistent predictions over time. We optimize this objective with points from the sampling distribution, effectively reducing the sampling drift observed in prior empirical works. Theoretically, we show that CP implies that we are sampling from the reverse of some diffusion process. Together with the assumption that the network has learned the score correctly in a subset of the domain, we can prove that CP (together with conservativity of the vector field) implies that the score is learned correctly everywhere - in the limit where the error approaches zero. Empirically, we use our objective to obtain state-of-the-art for CIFAR-10 and baseline improvements on AFHQ and FFHQ.
There are limitations of our method and several directions for future work. The proposed regularization increases the training time. It would be interesting to explore how to enforce CP in more effective ways in future work. Further, our method does not test nor enforce that the induced vector-field is conservative, which is a key theoretical assumption. Our method guarantees only indirectly improve the performance in the samples from the learned distribution by enforcing some invariant. Finally, our theoretical result holds in the limit where the error of our regularized objective approaches zero and it would be meaningful to theoretically study also the constant-error regime.
This research has been supported by NSF Grants CCF 1763702, AF 1901292, CNS 2148141, Tripods CCF 1934932, IFML CCF 2019844, the Texas Advanced Computing Center (TACC) and research gifts by Western Digital, WNCG IAP, UT Austin Machine Learning Lab (MLL), Cisco and the Archie Straiton Endowed Faculty Fellowship. Giannis Daras has been supported by the Onassis Fellowship, the Bodossaki Fellowship and the Leventis Fellowship. Constantinos Daskalakis has been supported by NSF Awards CCF-1901292, DMS-2022448 and DMS2134108, a Simons Investigator Award, the Simons Collaboration on the Theory of Algorithmic Fairness and a DSTA grant.
[1] Namrata Anand and Tudor Achim. Protein structure and sequence generation with equivariant denoising diffusion probabilistic models. arXiv preprint arXiv:2205.15019, 2022.
[2] Brian D.O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313-326, 1982.
A Proof of Theorem 3.2
In Section A.1 we present a proof overview, in Section A.2 we present some preliminaries to the proof, in Section A.3 we include the proof, with proofs of some lemmas omitted and in the remaining sections we prove these lemmas.
A.1 Proof overview
We start with the first part of the theorem. We assume that  $h$  satisfies Properties 1 and 2 and we will show that  $h$  is defined by Eq. (2) for some distribution  $p_0$  (while the other direction in the equivalence follows trivially from the definitions of these properties). Motivated by Eq. (6), define the function  $s\colon \mathbb{R}^d\times (0,1]$  according to
$$
s (x, t) = \frac {h (x , t) - x}{\sigma_ {t} ^ {2}}. \tag {10}
$$
We will first show that  $s$  satisfies the partial differential equation
$$
\frac {\partial s}{\partial t} = g (t) ^ {2} \left(J _ {s} s + \frac {1}{2} \triangle s\right), \tag {11}
$$
where  $J_{s}\in \mathbb{R}^{d\times d}$  is the Jacobian of  $s$ ,  $(J_{s})_{ij} = \frac{\partial s_{i}}{x_{j}}$  and each coordinate  $i$  of  $\triangle s\in \mathbb{R}^d$  is the Laplacian of coordinate  $i$  of  $s$ ,  $(\triangle s)_i = \sum_{j = 1}^n\frac{\partial^2s_j}{\partial x_j^2}$ . In order to obtain Eq. (11), first, we use a generalization of Ito's lemma, which states that for an SDE
$$
d x _ {t} = \mu \left(x _ {t}, t\right) d t + g (t) d \bar {B} _ {t} x \tag {12}
$$
and for  $f\colon \mathbb{R}^d\times [0,1]\to \mathbb{R}^d,f(x_t,t)$  satisfies the SDE
$$
d f (x _ {t}, t) = \left(\frac {\partial f}{\partial t} + J _ {f} \mu - \frac {g (t) ^ {2}}{2} \triangle f\right) d t + g (t) J _ {f} d \bar {B} _ {t}.
$$
If  $f$  is a reverse-Martingale then the term that multiplies  $dt$  has to equal zero, namely,
$$
\frac {\partial f}{\partial t} + J _ {f} \mu - \frac {g (t) ^ {2}}{2} \triangle f = 0.
$$
By Lemma 3.1,  $h(x_{t}, t)$  is a reverse Martingale, therefore we can substitute  $f = h$  and substitute  $\mu = -g(t)^{2}s$  according to Eq. (7), to deduce that
$$
\frac {\partial h}{\partial t} - g (t) ^ {2} J _ {h} s - \frac {g (t) ^ {2}}{2} \triangle h = 0.
$$
Substituting  $h(x,t) = \sigma_t^2 s(x,t) + x$  according to Eq. (6) yields Eq. (11) as required.
Next, we show that any  $s'$  that is the score-function (i.e. gradient of log probability) of some diffusion process that follows the SDE Eq. (4), also satisfies Eq. (11). To obtain this, one can use the Fokker-Planck equation, whose special case states that the density function  $p(x,t)$  of any stochastic process that satisfies the SDE Eq. (4) satisfies the PDE
$$
\frac {\partial p}{\partial t} = \frac {g (t) ^ {2}}{2} \triangle p
$$
where  $\triangle$  corresponds to the Laplacian operator. Using this one can obtain a PDE for  $\nabla_{x}\log p$  which happens to be exactly Eq. (11) if the process is defined by Eq. (4).
Next, we use Property 2 to deduce that there exists some densities  $p(\cdot ,t)$  for  $t\in [0,1]$  such that
$$
s (x, t) = \frac {h (x , t) - x}{\sigma_ {t} ^ {2}} = \nabla_ {x} \log p (x, t).
$$
Denote by  $p'(x, t)$  the score function of the diffusion process that is defined by the SDE of Eq. (4) with the initial condition that  $p(x, 0) = p'(x, 0)$  for all  $x$ . Denote by  $s'(x, t) = \nabla_x \log p'(x, t)$  the score function of  $p'$ . As we proved above, both  $s$  and  $s'$  satisfy the PDE Eq. (11) and the same initial
condition at  $t = 0$ . By the uniqueness of the PDE, it holds that  $s(x,t) = s'(x,t)$  for all  $t \geq t_0$ . Denote by  $h^*$  the function that satisfies Eq. (2) with the initial condition  $x_0 \sim p_0$ . By Eq. (6),
$$
s ^ {\prime} (x, t) = \frac {h ^ {*} (x , t) - x}{\sigma_ {t} ^ {2}}.
$$
By Eq. (10) and since  $s = s'$ , it follows that  $h = h^*$  and this is what we wanted to prove.
We proceed with proving part 2 of the theorem. We use the notion of an analytic function on  $\mathbb{R}^d$ : that is a function  $f\colon \mathbb{R}^d\to \mathbb{R}$  such that at any  $x_0\in \mathbb{R}^d$ , the Taylor series of  $f$  centered at  $x_0$  converges for all  $x\in \mathbb{R}^d$  to  $f(x)$ . We use the property that an analytic function is uniquely determined by its value on any open subset: If  $f$  and  $g$  are analytic functions that identify in some open subset  $U\subset \mathbb{R}^d$  then  $f = g$  everywhere. We prove this statement in the remainder of this paragraph, as follows: Represent  $f$  and  $g$  as Taylor series around some  $x_0\in U$ . The Taylor series of  $f$  and  $g$  identify: indeed, these series are functions of the derivatives of  $f$  and  $g$  which are functions of only the values in  $U$ . Since  $f$  and  $g$  equal their Taylor series, they are equal.
Next, we will show that for any diffusion process that is defined by Eq. (4), the probability density of  $p(x,t_0)$  at any time  $t_0 > 0$  is analytic as a function of  $x$ . Recall that the distribution of  $x_0$  is defined in Eq. (4) as  $p_0$  and it holds that the distribution of  $x_{t_0}$  is obtained from  $p_0$  by adding a Gaussian noise  $\mathcal{N}(0,\sigma_t^2 I)$  and its density at any  $x$  equals
$$
p (x, t _ {0}) = \int_ {a \in \mathbb {R} ^ {d}} \frac {1}{\sqrt {2 \pi} \sigma_ {t _ {0}}} \exp \left(- \frac {(x - a) ^ {2}}{2 \sigma_ {t} ^ {2}}\right) d p _ {0} (a).
$$
Since the function  $\exp(-\left(x - a\right)^2 / \left(2\sigma_t^2\right))$  is analytic, one could deduce that  $p(x, t_0)$  is also analytic. Further,  $p(x, t_0) > 0$  for all  $x$  which implies that there is no singularity for  $\log p(x, t_0)$  which can be used to deduce that  $\log p(x, t_0)$  is also analytic and further that  $\nabla_x \log p(x, t_0)$  is analytic as well.
We use the first part of the theorem to deduce that  $s$  is the score function of some diffusion process hence it is analytic. By assumption,  $s$  identifies with some target score function  $s^*$  in some open subset  $U \subseteq \mathbb{R}^d$  at some  $t_0$ , which, by the fact that  $s(x, t_0)$  and  $s^*(x, t_0)$  are analytic, implies that  $s(x, t_0) = s^*(x, t_0)$  for all  $x$ . Finally, since  $s$  and  $s^*$  both satisfy the PDE Eq. (11) and they satisfy the same initial condition at  $t_0$ , it holds that by uniqueness of the PDE  $s(x, t) = s^*(x, t)$  for all  $x$  and  $t$ .
A.2 Preliminaries
Preliminaries on diffusion processes In the next definition we define for a function  $F\colon \mathbb{R}^d\to \mathbb{R}^d$  its Jacobian  $J_{F}$ , its divergence  $\nabla \cdot F$  and its Laplacian  $\triangle F$  that is computed separately on each coordinate of  $F$ :
Definition A.1. Given a function  $F = (f_{1},\ldots ,f_{n})\colon \mathbb{R}^{d}\to \mathbb{R}^{d}$ , denote by  $J_{F}\colon \mathbb{R}^{d}\to \mathbb{R}^{d\times d}$  its Jacobian:
$$
\left(J _ {F}\right) _ {i j} = \frac {\partial f _ {i} (x)}{\partial x _ {j}}.
$$
The divergence of  $F$  is defined as
$$
\nabla \cdot F (x) := \sum_ {i = 1} ^ {n} \frac {\partial f _ {i} (x)}{\partial x _ {i}}.
$$
Denote by  $\triangle F\colon \mathbb{R}^d\to \mathbb{R}^d$  the function whose  $i$ th entry is the Laplacian of  $f_{i}$ :
$$
(\triangle F (x)) _ {i} = \sum_ {j = 1} ^ {n} \frac {\partial^ {2} f _ {i} (x)}{\partial x _ {j} ^ {2}}.
$$
If  $F$  is a function of both  $x \in \mathbb{R}^d$  and  $t \in \mathbb{R}$ , then  $J_F$ ,  $\triangle f$  and  $\nabla \cdot F$  correspond to  $F$  as a function of  $x$ , whereas  $t$  is kept fixed. In particular,
$$
(J _ {F} (x, t)) _ {i j} = \frac {\partial f _ {i} (x , t)}{\partial x _ {j}}, \quad (\triangle F (x, t)) _ {i} = \sum_ {j = 1} ^ {n} \frac {\partial^ {2} f _ {i} (x , t)}{\partial x _ {j} ^ {2}}, \quad \nabla \cdot F = \sum_ {i = 1} ^ {n} \frac {\partial f _ {i} (x , t)}{\partial x _ {i}}.
$$
We use the celebrated Ito's lemma and some of its immediate generalizations:
Lemma A.2 (Ito's Lemma). Let  $x_{t}$  be a stochastic process  $x_{t} \in \mathbb{R}^{d}$ , that is defined by the following SDE:
$$
d x _ {t} = \mu (x _ {t}, t) d t + g (t) d B _ {t},
$$
where  $B_{t}$  is a standard Brownian motion. Let  $f\colon \mathbb{R}^d\times \mathbb{R}\to \mathbb{R}$ . Then,
$$
d f (x _ {t}, t) = \left(\frac {d f}{d t} + \nabla_ {x} f ^ {\top} \mu (x _ {t}, t) + \frac {g (t) ^ {2}}{2} \triangle f\right) d t + g (t) \nabla_ {x} f ^ {\top} d B _ {t}.
$$
Further, if  $F\colon \mathbb{R}^d\times \mathbb{R}\to \mathbb{R}^d$  is a multi-valued function, then