SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Suppose we had access to the ground-truth scores  $\mathbf{s}_{\mathrm{true}}^{(1)}(\mathbf{x},t)$  and  $\mathbf{s}_{\mathrm{true}}^{(2)}(\mathbf{x},t)$  of the forward processes for the base and expert models (i.e. no approximation error). Then they are equal to  $\nabla_{\mathbf{x}}\log p_t^{(1)}(\mathbf{x})$  and  $\nabla_{\mathbf{x}}\log p_t^{(2)}(\mathbf{x})$ , respectively. If we also assume  $p_t^{(1)}(\mathbf{x})$  and  $p_t^{(2)}(\mathbf{x})$  are continuously differentiable, we have that, by Definition 4.2, the diffusion models are conservative for each  $t$ . Thus, their difference is also conservative, i.e. the gradient of a continuously differentiable function.
Hence, by the Fundamental Theorem for Line Integrals (Th. 7.2 in [35]), there exists a unique  $\rho$  satisfying  $\nabla_{\mathbf{x}}\rho (\mathbf{x},t) = \mathbf{s}_{\mathrm{true}}^{(2)}(\mathbf{x},t) - \mathbf{s}_{\mathrm{true}}^{(1)}(\mathbf{x},t)$ , up to an additive constant, given by the line integral
$$
\rho (\mathbf {x}, t) = \int_ {\mathbf {x} _ {\text {r e f}}} ^ {\mathbf {x}} \left[ \mathbf {s} _ {\text {t r u e}} ^ {(2)} \left(\mathbf {x} ^ {\prime}, t\right) - \mathbf {s} _ {\text {t r u e}} ^ {(1)} \left(\mathbf {x} ^ {\prime}, t\right) \right] \cdot d \mathbf {x} ^ {\prime}, \tag {10}
$$
where  $\mathbf{x}_{\mathrm{ref}}$  is some arbitrary reference point, and the line integral is path-independent.
In practice, however, we cannot guarantee the absence of approximation errors, nor that the diffusion models are conservative.
4.2 Relative Reward Function of Two Diffusion Models
To get around the possibility that  $\mathbf{s}^{(1)}$  and  $\mathbf{s}^{(2)}$  are not conservative, we may instead look for the conservative field best approximating  $\mathbf{s}^{(2)}(\mathbf{x},t) - \mathbf{s}^{(1)}(\mathbf{x},t)$  in  $L^2 (\mathbb{R}^n,\mathbb{R}^n)$  (i.e. the space of square-integrable vector fields, endowed with the  $L^2$  norm). Using a well-known fundamental result on uniqueness of projections in  $L^2$  (Th. C.8), we obtain the following:
Proposition 4.3 (Optimal Relative Reward Gradient). Let  $\mathbf{s}_{\phi}^{(1)}$  and  $\mathbf{s}_{\Theta}^{(2)}$  be any two diffusion models and  $t \in (0, T]$ , with the assumption that  $\mathbf{s}_{\Theta}^{(2)}(\cdot, t) - \mathbf{s}_{\phi}^{(1)}(\cdot, t)$  is square-integrable. Then there exists a unique vector field  $\mathbf{h}_t$  given by
$$
\mathbf {h} _ {t} = \underset {\mathbf {f} \in \overline {{\operatorname {C o n s} (\mathbb {R} ^ {n})}}} {\operatorname {a r g m i n}} \int_ {\mathbb {R} ^ {n}} | | \mathbf {f} (\mathbf {x}) - \left(\mathbf {s} _ {\Theta} ^ {(2)} (\mathbf {x}, t) - \mathbf {s} _ {\phi} ^ {(1)} (\mathbf {x}, t)\right) | | _ {2} ^ {2} d \mathbf {x} \tag {11}
$$
where  $\overline{\mathrm{Cons}(\mathbb{R}^n)}$  denotes the closed span of gradients of smooth  $W^{1,2}$  potentials. Furthermore, for any  $\varepsilon > 0$ , there is a smooth, square-integrable potential  $\Phi$  with a square-integrable gradient satisfying:
$$
\int_ {\mathbb {R} ^ {n}} \left| \left| \nabla_ {\mathbf {x}} \Phi (\mathbf {x}) - \mathbf {h} _ {t} (\mathbf {x}) \right| \right| _ {2} ^ {2} d \mathbf {x} <   \varepsilon \tag {12}
$$
We call such an  $\mathbf{h}_t$  the optimal relative reward gradient of  $\mathbf{s}_{\phi}^{(1)}$  and  $\mathbf{s}_{\Theta}^{(2)}$  at time  $t$ .
For proof of Proposition 4.3 see Appendix C.4. It is important to note that for the projection to be well-defined, we required an assumption regarding the integrability of the diffusion models. Without this assumption, the integral would simply diverge.
The result in Proposition 4.3 tells us that we can get arbitrarily close to the optimal relative reward gradient using scalar potentials' gradients. Therefore, we may finally define the central notion in this paper:
Algorithm 1: Relative reward function training.
Input: Base  $\mathbf{s}^{(1)}$  and expert  $\mathbf{s}^{(2)}$  diffusion models, dataset  $\mathcal{D}$  number of iterations  $I$    
Output: Relative reward estimator  $\rho_{\theta}$    
Initialize reward estimator parameters  $\theta$    
for  $j\in \{1,\dots,I\}$  do Sample batch:  $\mathbf{X}_0 = [\mathbf{x}_0^{(1)},\dots,\mathbf{x}_0^{(N)}]$  from  $\mathcal{D}$  Sample times:  $\mathbf{t} = [t_1,\dots,t_N]$  independently in  $\mathcal{U}(0,T]$  Sample forward process:  $\mathbf{X}_{\mathbf{t}}\gets [\mathbf{x}_{t_1}^{(1)},\dots,\mathbf{x}_{t_N}^{(N)}]$  Take an optimization step on  $\theta$  according to  $\hat{L}_{\mathrm{RRF}}(\theta) =$ $\frac{1}{N}\sum_{i = 1}^{N}||\nabla_{\mathbf{x}}\rho_{\theta}(\mathbf{x}_{t_i}^{(i)},t_i) - (\mathbf{s}_{\Theta}^{(2)}(\mathbf{x}_{t_i}^{(i)},t_i) - \mathbf{s}_{\phi}^{(1)}(\mathbf{x}_{t_i}^{(i)},t_i))||_2^2$  end
Figure 2: Learned rewards for base and expert diffusion models from Stable Diffusion (Sec. 5). Prompts from the I2P dataset.
Definition 4.4 ( $\varepsilon$ -Relative Reward Function). For an  $\varepsilon > 0$ , an  $\varepsilon$ -relative reward function of diffusion models  $\mathbf{s}_{\phi}^{(1)}$  and  $\mathbf{s}_{\Theta}^{(2)}$  is a function  $\rho: \mathbb{R}^n \times [0, T] \to \mathbb{R}$  such that
$$
\forall t \in (0, T ]: \int_ {\mathbb {R} ^ {n}} | | \nabla_ {\mathbf {x}} \rho (\mathbf {x}, t) - \mathbf {h} _ {t} (\mathbf {x}) | | _ {2} ^ {2} d \mathbf {x} <   \varepsilon \tag {13}
$$
where  $\mathbf{h}_t$  denotes the optimal relative reward gradient of  $\mathbf{s}_{\phi}^{(1)}$  and  $\mathbf{s}_{\Theta}^{(2)}$  at time  $t$ .
4.3 Extracting Reward Functions
We now set out to actually approximate the relative reward function  $\rho$ . Definition 4.4 naturally translates into an  $L^2$  training objective for learning  $\rho$ :
$$
L _ {\mathrm {R R F}} (\theta) = \mathbb {E} _ {t \sim \mathcal {U} [ 0, T ], \mathbf {x} _ {t} \sim p _ {t}} \left[ | | \nabla_ {\mathbf {x}} \rho_ {\theta} (\mathbf {x} _ {t}, t) - \left(\mathbf {s} _ {\Theta} ^ {(2)} (\mathbf {x} _ {t}, t) - \mathbf {s} _ {\phi} ^ {(1)} (\mathbf {x} _ {t}, t)\right) | | _ {2} ^ {2} \right] \tag {14}
$$
where  $p_t$  denotes the marginal at time  $t$  of the forward noising process.
We optimize this objective via Empirical Risk Minimization and Stochastic Gradient Descent. See Algorithm 1 for a version assuming access to the diffusion models and their training datasets, and Algorithm 2 in Appendix D for one which does not assume access to any pre-existing dataset. Our method requires no access to the environment or to a simulator. Our algorithm requires computing a second-order mixed derivative  $D_{\theta}(\nabla_{\mathbf{x}}\rho (\mathbf{x},t))\in \mathbb{R}^{m\times n}$  (where  $m$  is the number of parameters  $\theta$ ), for which we use automatic differentiation in PyTorch [46].
Recovering Per-Time-Step Rewards. We can parameterize  $\rho$  using a single-time-step neural network  $g_{\theta}(s,a,t)$  as  $\rho (\tau_t,t) = \frac{1}{N}\sum_{i = 1}^N g_\theta (s_t^i,a_t^i,t)$ , where  $N$  is the horizon and  $\tau_{t}$  denotes a trajectory at diffusion timestep  $t$ , and  $s_t^i$  and  $a_{t}^{i}$  denote the  $i^{\mathrm{th}}$  state and action in  $\tau_{t}$ . Then,  $g_{\theta}(s,a,0)$  predicts a reward for a state-action pair  $(s,a)$ .
In this section, we conduct empirical investigations to analyze the properties of relative reward functions in practice. Our experiments focus on three main aspects: the alignment of learned reward functions with the goals of the expert agent, the performance improvement achieved through steering the base model using the learned reward, and the generalizability of learning reward-like functions to domains beyond decision-making (note that the relative reward function is defined for any pair of diffusion models). For details on implementation specifics, computational requirements, and experiment replication instructions, we refer readers to the Appendix.
5.1 Learning Correct Reward Functions from Long-Horizon Plans
Maze2D [12] features various environments which involve controlling the acceleration of a ball to navigate it towards various goal positions in 2D mazes. It is suitable for evaluating reward learning, as it requires effective credit assignment over extended trajectories.
Figure 3: Heatmaps of the learned relative reward in Maze2D.  $\star$  denotes the ground-truth goal position. For more examples, see Appendix E.3.
Implementation. To conduct our experiments, we generate multiple datasets of trajectories in each of the 2D environments, following the data generation procedure from D4RL [12], except sampling start and goal positions uniformly at random (as opposed to only at integer coordinates).
For four maze environments with different wall configurations (depicted in Figure 3), we first train a base diffusion model on a dataset of uniformly sampled start and goal positions, hence representing undirected behavior. For each environment, we then train eight expert diffusion models on datasets with fixed goal positions. As there are four maze configurations and eight goal positions per maze configuration, we are left with 32 expert models in total. For each of the 32 expert models, we train a relative reward estimator  $\rho_{\theta}$ , implemented as an MLP, via gradient alignment, as described in Alg. 1. We repeat this across 5 random seeds.
Discriminability Results. For a quantitative evaluation, we train a logistic regression classifier per expert model on a balanced dataset of base and expert trajectories. Its objective is to label trajectories as base or expert using only the predicted reward as input. We repeat this process for each goal position with 5 different seeds, and average accuracies across seeds. The achieved accuracies range from  $65.33\%$  to  $97.26\%$ , with a median of  $84.49\%$  and a mean of  $83.76\%$ . These results demonstrate that the learned reward effectively discriminates expert trajectories.
Visualization of Learned Reward Functions. To visualize the rewards, we use our model to predict rewards for fixed-length sub-trajectories in the base dataset. We then generate a 2D heatmap by averaging the rewards of all sub-trajectories that pass through each grid cell. Fig. 3 displays some of these heatmaps, with more examples given in App. F.
Results. We observe in Fig. 3 that the network accurately captures the rewards, with peaks occurring at the true goal position of the expert dataset in  $78.75\% \pm 8.96\%$  of the cases for simpler mazes (maze2d-open-v0 and maze2d-umaze-v0), and  $77.50\% \pm 9.15\%$  for more advanced mazes (maze2d-medium-v1 and maze2d-large-v1). Overall, the network achieves an average success rate of  $78.12\% \pm 6.40\%$ . We further verify the correctness of the learned reward functions by retraining agents in Maze2D using the extracted reward functions (see Appendix E.2.1).
Sensitivity to Dataset Size. Since training diffusion models typically demands a substantial amount of data, we investigate the sensitivity of our method to dataset size. Remarkably, we observe that the performance only experiences only a slight degradation even with significantly smaller sizes. For complete results, please refer to Appendix E.3.
5.2 Steering Diffusion Models to Improve their Performance
Having established the effectiveness of relative reward functions in recovering expert goals in the low-dimensional Maze2D environment, we now examine their applicability in higher-dimensional control tasks. Specifically, we evaluate their performance in the HalfCheetah, Hopper, and Walker-2D environments from the D4RL offline locomotion suite [12]. These tasks involve controlling a multi-degree-of-freedom 2D robot to move forward at the highest possible speed. We assess the learned reward functions by examining whether they can enhance the performance of a weak base model when used for classifier-guided steering. If successful, this would provide evidence that the learned relative rewards can indeed bring the base trajectory distribution closer to the expert distribution.
Implementation. In these locomotion environments, the notion of reward is primarily focused on moving forward. Therefore, instead of selecting expert behaviors from a diverse set of base behaviors, our reward function aims to guide a below-average-performing base model toward improved performance. Specifically, we train the base models using the medium-replay datasets from D4RL, which yield low rewards, and the expert models using the expert datasets, which yield high rewards.
Figure 4: The 3 images with the highest learned reward in their batch ("safe"), and 3 with the lowest reward ("unsafe", blurred), respectively.
Table 1: Diffuser performance with different steering methods, on 3 RL environments and for a low-performance base model (top) and a medium-performance model (bottom).
Environment	Unsteered	Discriminator	Reward (Ours)
Halfcheetah	30.38 ± 0.38	30.41 ± 0.38	31.65 ± 0.32
Hopper	24.67 ± 0.92	25.12 ± 0.95	27.04 ± 0.90
Walker2d	28.20 ± 0.99	27.98 ± 0.99	38.40 ± 1.02
Mean	27.75	27.84	32.36
Halfcheetah	59.41 ± 0.87	70.79 ± 1.92	69.32 ± 0.8
Hopper	58.8 ± 1.01	59.42 ± 2.23	64.97 ± 1.15
Walker2d	96.12 ± 0.92	96.75 ± 1.91	102.05 ± 1.15
Mean	71.44	75.65	78.78
The reward functions are then fitted using gradient alignment as described in Algorithm 1. Finally, we employ classifier guidance (Section 3) to steer each base model using the corresponding learned reward function.
Results. We conduct 512 independent rollouts of the base diffusion model steered by the learned reward, using various guidance scales  $\omega$  (Eq. 5). We use the unsteered base model as a baseline. We also compare our approach to a discriminator with the same architecture as our reward function, trained to predict whether a trajectory originates from the base or expert dataset. We train our models with 5 random seeds, and run the 512 independent rollouts for each seed. Steering the base models with our learned reward functions consistently leads to statistically significant performance improvements across all three environments. Notably, the Walker2D task demonstrates a  $36.17\%$  relative improvement compared to the unsteered model. This outcome suggests that the reward functions effectively capture the distinctions between the two diffusion models. See Table 1 (top three rows). We further conducted additional experiments in the Locomotion domains in which we steer a new medium-performance diffusion model unseen during training, using the relative reward function learned from the base diffusion model and the expert diffusion model. We observed in Table 1 (bottom three rows) that our learned reward function significantly improves performance also in this scenario.
5.3 Learning a Reward-Like Function for Stable Diffusion
While reward functions are primarily used in sequential decision-making problems, we propose a generalization to more general domains through the concept of relative reward functions, as discussed in Section 4. To empirically evaluate this generalization, we focus on one of the domains where diffusion models have demonstrated exceptional performance: image generation. Specifically, we examine Stable Diffusion [54], a widely used  $859\mathrm{m}$ -parameter diffusion model for general-purpose image generation, and Safe Stable Diffusion [59], a modified version of Stable Diffusion designed to mitigate the generation of inappropriate images.
Models. The models under consideration are latent diffusion models, where the denoising process occurs in a latent space and is subsequently decoded into an image. These models employ classifier-free guidance [22] during sampling and can be steered using natural language prompts, utilizing CLIP embeddings [49] in the latent space. Specifically, Safe Stable Diffusion [59] introduces modifications to the sampling loop of the open-source model proposed by Rombach et al. [54], without altering the model's actual weights. In contrast to traditional classifier-free guidance that steers samples toward a given prompt, the modified sampler of Safe Stable Diffusion also directs samples away from undesirable prompts [59, Sec. 3].
Prompt Dataset. To investigate whether our reward networks can detect a "relative preference" of Safe Stable Diffusion over the base Stable Diffusion model for harmless images, we use the I2P prompt dataset introduced by Schramowski et al. [59]. This dataset consists of prompts specifically designed to deceive Stable Diffusion into generating imagery with unsafe content. However, we use the dataset to generate sets of image embeddings rather than actual images, which serve as training data for our reward networks. A portion of the generated dataset containing an equal number of base and expert samples is set aside for model evaluation.
Separating Image Distributions. Despite the complex and multimodal nature of the data distribution in this context, we observe that our reward networks are capable of distinguishing between base and expert images with over  $90\%$  accuracy, despite not being explicitly trained for this task. The reward histogram is visualized in Figure 2.
Qualitative Evaluation. We find that images that receive high rewards correspond to safe content, while those with low rewards typically contain unsafe or disturbing material, including hateful or offensive imagery. To illustrate this, we sample batches from the validation set, compute the rewards for each image, and decode the image with the highest reward and the one with the lowest reward from each batch. Considering the sensitive nature of the generated images, we blur the latter set as an additional safety precaution. Example images can be observed in Figure 4.
To the best of our knowledge, our work introduces the first method for extracting relative reward functions from two diffusion models. We provide theoretical justification for our approach and demonstrate its effectiveness in diverse domains and settings. We expect that our method has the potential to facilitate the learning of reward functions from large pre-trained models, improving our understanding and the alignment of the generated outputs. It is important to note that our experiments primarily rely on simulated environments, and further research is required to demonstrate its applicability in real-world scenarios.
This work was supported by the Royal Academy of Engineering (RF\201819\18\163). F.N. receives a scholarship from Fundação Estudar. F.N. also used TPUs granted by the Google TPU Research Cloud (TRC) in the initial exploratory stages of the project.
We would like to thank Luke Melas-Kyriazi, Prof. Varun Kanade, Yizhang Lou, Ruining Li and Eduard Oravkin for proofreading versions of this work.
F.N. would also like to thank Prof. Stefan Kiefer for the support during the project, and Michael Janner for the productive conversations in Berkeley about planning with Diffusion Models.
We use the following technologies and repositories as components in our code: PyTorch [46], NumPy [18], Diffuser [25], D4RL [12], HuggingFace Diffusers [67] and LucidRains's Diffusion Models in Pytorch repository.
[67] von Platen, P., Patil, S., Lozhkov, A., Cuenca, P., Lambert, N., Rasul, K., Davaadorj, M. and Wolf, T. [2022], 'Diffusers: State-of-the-art diffusion models', https://github.com/huggingface/diffusers.
[68] Wang, Z., Hunt, J. J. and Zhou, M. [2022], 'Diffusion policies as an expressive policy class for offline reinforcement learning', arXiv preprint arXiv:2208.06193.
[69] Zhong, Z., Rempe, D., Xu, D., Chen, Y., Veer, S., Che, T., Ray, B. and Pavone, M. [2022], 'Guided conditional diffusion for controllable traffic simulation', arXiv preprint arXiv:2210.17366.
[70] Ziebart, B. D. [2010], Modeling purposeful adaptive behavior with the principle of maximum causal entropy, Carnegie Mellon University.
[71] Ziebart, B. D., Maas, A. L., Bagnell, J. A., Dey, A. K. et al. [2008], Maximum entropy inverse reinforcement learning., in 'Aaai', Vol. 8, Chicago, IL, USA, pp. 1433-1438.
Appendix A More Details on Diffusion Models
A.1 Diffusion Models in Continuous Time
At a high level, diffusion models work by adding noise to data  $\mathbf{x}$ , and then learning to denoise it.
The seminal paper on diffusion models [61] and the more recent work of Ho et al. [21] describe this in discrete time. Song et al. [65] show a direct correspondence between the aforementioned formulations of denoising diffusion models and an existing line of work on score-based generative modeling [63], and propose a continuous-time framework leveraging (Ito) Stochastic Differential Equations [36, 45, 58] to unify both methods. We give an overview of the continuous-time formulation, and defer to Appendix A.2 an account of the discrete case.
The forward noising process in continuous time is given by:
$$
d \mathbf {x} _ {t} = \mathbf {f} \left(\mathbf {x} _ {t}, t\right) d t + g (t) d \mathbf {w} _ {t} \tag {15}
$$
where  $f$  is a function that is Lipschitz,  $\mathbf{w}$  is a standard Brownian Motion [36] and  $g$  is a "continuous noise schedule" (c.f. A.2). A continuous version of the processes in A.2 [61, 21] is recovered by setting  $g(s) = \sqrt{\beta_s}$  and  $\mathbf{f}(\mathbf{x}, s) = -\frac{1}{2}\beta_s\mathbf{x}$ . The resulting SDE resembles an Ornstein-Uhlenbeck process, which is known to converge in geometric rate to a  $\mathcal{N}(0, I)$  distribution. This justifies the choice of distribution of  $\mathbf{x}_T$  in A.2.
Song et al. [65] then use a result of Anderson [4] to write the SDE satisfied by the reverse process of 2, denoted  $\bar{\mathbf{x}}_t$ , as:
$$
d \bar {\mathbf {x}} _ {t} = \left[ \mathbf {f} \left(\bar {\mathbf {x}} _ {t}, t\right) - g (t) ^ {2} \nabla_ {\mathbf {x}} \log p _ {t} \left(\bar {\mathbf {x}} _ {t}\right) \right] d t + g (t) d \bar {\mathbf {w}} _ {t} \tag {16}
$$
Here  $p_t(\mathbf{x})$  denotes the marginal density function of the forward process  $\mathbf{x}_t$ , and  $\bar{\mathbf{w}}$  is a reverse Brownian motion (for further details on the latter, see [4]). The learning step is then to learn an approximation  $\mathbf{s}_{\Theta}(\mathbf{x}_t,t)$  to  $\nabla_{\mathbf{x}}\log p_t(\mathbf{x}_t)$  (the score of the distribution of  $\mathbf{x}_t$ ). This is shown by Song et al. [65] to be equivalent to the noise model  $\epsilon_{\Theta}$  in A.2. The sampling step consists of taking  $\bar{\mathbf{x}}_T\sim \mathcal{N}(0,I)$  for some  $T > 0$ , and simulating 16 backwards in time to arrive at  $\bar{\mathbf{x}}_0\approx \mathbf{x}_0$ .
One of the advantages of this perspective is that the sampling loop can be offloaded to standard SDE solvers, which can more flexibly trade-off e.g. computation time for performance, compared to hand-designed discretizations [32].
A.2 Discrete-Time Diffusion Models
The seminal paper on diffusion models [61] and the more recent work of Ho et al. [21] formulate diffusion models in discrete time. The forward noising is a Markov Chain starting from an uncorrupted data point  $\mathbf{x}_0$ , and repeatedly applying a (variance-preserving) Gaussian kernel to the data:
$$
q _ {t} \left(\mathbf {x} _ {t + 1} \mid \mathbf {x} _ {t}\right) \sim \mathcal {N} \left(\sqrt {1 - \beta_ {t}} \mathbf {x} _ {t}, \beta_ {t} I\right)
$$
The coefficients  $\beta_{t}$  are known as the noise schedule of the diffusion model, and can be chosen to regulate the speed with which noise is added to the data [32].
The modeling step then aims to revert the noising process by learning a backward kernel
$$
p _ {\theta} \left(\mathbf {x} _ {t - 1} \mid \mathbf {x} _ {t}\right) \sim \mathcal {N} \left(\boldsymbol {\mu} _ {\theta} \left(\mathbf {x} _ {t}, t\right), \sigma_ {t} ^ {2}\right) \tag {17}
$$
The seminal paper of Sohl-Dickstein et al. [61] proposes a variational loss for training  $\mu_{\theta}$ , based on a maximum-likelihood framework. Subsequently, Ho et al. [21] proposed a new parametrization of  $\mu_{\theta}$ , as well as a much simpler loss function. Their loss function is
$$
L _ {\text {s i m p l e}} (\theta) := \mathbb {E} _ {t, \mathbf {x} _ {0}, \epsilon} \left[ \left\| \boldsymbol {\epsilon} - \boldsymbol {\epsilon} _ {\theta} \left(\sqrt {\bar {\alpha} _ {t}} \mathbf {x} _ {0} + \sqrt {1 - \bar {\alpha} _ {t}} \boldsymbol {\epsilon}, t\right) \right\| ^ {2} \right] \tag {18}
$$