SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Puheng Li**
puhengli@stanford.edu
Zhong Li*†
lzhong@microsoft.com
Huishuai Zhang†
Jiang Bian
jiabia@microsoft.com
Diffusion models are a class of generative models that serve to establish a stochastic transport map between an empirically observed, yet unknown, target distribution and a known prior. Despite their remarkable success in real-world applications, a theoretical understanding of their generalization capabilities remains underdeveloped. This work embarks on a comprehensive theoretical exploration of the generalization attributes of diffusion models. We establish theoretical estimates of the generalization gap that evolves in tandem with the training dynamics of score-based diffusion models, suggesting a polynomially small generalization error  $(O(n^{-2/5} + m^{-4/5}))$  on both the sample size  $n$  and the model capacity  $m$ , evading the curse of dimensionality (i.e., not exponentially large in the data dimension) when early-stopped. Furthermore, we extend our quantitative analysis to a data-dependent scenario, wherein target distributions are portrayed as a succession of densities with progressively increasing distances between modes. This precisely elucidates the adverse effect of "modes shift" in ground truths on the model generalization. Moreover, these estimates are not solely theoretical constructs but have also been confirmed through numerical simulations. Our findings contribute to the rigorous understanding of diffusion models' generalization properties and provide insights that may guide practical applications.
As an emerging family of deep generative models, diffusion models (DMs; [16, 45]) have experienced a surge in popularity, owing to their unparalleled performance in a wide range of applications ([22, 39, 68, 15, 3, 35, 4, 60, 33, 69, 64, 63, 53]). This has led to notable commercial successes, such as DALL-E ([34]), Imagen ([38]), and Stable Diffusion ([36]). Mathematically, diffusion models learn an unknown underlying distribution through a two-stage process: (i) first, successively and gradually injecting random noises (forward process); (ii) then reversing the forward process through denoising for sampling purposes (reverse process). To achieve this, an equivalent formulation of diffusion models called score-based generative models (SGMs; [50, 52]) is employed. SGMs implement the aforementioned two-stage process via the continuous dynamics represented by a joint group of coupled stochastic differential equations (SDEs) ([54, 20]).
*This work was done when Puheng Li was an undergraduate student at Peking University and a research intern at MSRA. The first two authors contributed equally to this work and are ordered alphabetically.
Despite their impressive empirical performance, the theoretical foundation of DMs/SGMs remains underexplored. Generally, fundamental theoretical questions can be categorized into several aspects. By considering machine learning models as mathematical function classes from certain spaces, one can identify three central aspects: approximation, optimization and generalization. At the forefront lies the generalization problem, which aims to characterize the learning error between the learned and ground truth distributions.
The development of generalization theory for diffusion models is pressing due to both theoretical and practical concerns:
- In theory, the generalization issues of generative modeling (or learning for distributions) may exhibit as the memorization phenomenon, if the modeled distribution is eventually trained to converge to the empirical distribution only associated with training samples. Intuitively, memorization arises from two reasons: (i) it is useful for the hypothesis space to be large enough to approximate highly complex underlying target distributions (universal convergence; [65]); (ii) the underlying distribution is unknown in practice, and one can only use a dataset with finite samples drawn from the target distribution. Rigorous mathematical characterizations of memorization are developed for bias potential models and GANs in [66] and [67], respectively. A natural question is, does a similar phenomenon occur for diffusion models? To answer this, a thorough investigation of generalization properties for DMs/SGMs is required.
- In practice, the generalization capability of diffusion models is also an essential requirement, as the memorization can lead to potential privacy and copyright risks when models are deployed. Similar to other generative models and large language models (LLMs) [5, 70, 19, 6], diffusion models can also memorize and leak training samples [5, 46], hence can be subsequently attacked using specific procedures and algorithms [28, 18, 62]. Although there are defense methods developed to meet privacy and copyright standards ([11, 14, 58]), these approaches are often heuristic, without providing sufficient quantitative understandings particularly on diffusion models. Therefore, a comprehensive investigation of the generalization foundation of diffusion models, including both theoretical and empirical aspects, is of utmost importance in improving principled tutorial guidance in practice.
The current work develops the generalization theory of diffusion models in a mathematically rigorous manner. Our main results include the following:
- We derive an upper bound of the generalization gap for diffusion models along the training dynamics. This result suggests, with early-stopping, the generalization error of diffusion models scales polynomially small on the sample size  $(O(n^{-2/5}))$  and the model capacity  $(O(m^{-4/5}))$ . Notably, the generalization error also escapes from the curse of dimensionality.
- This "uniform" bound is further extended to a data-dependent setting, where a sequence of unidimensional Gaussian mixtures distributions with an increasing modes' distance is considered as the ground truth. This result characterizes the effect of "modes shift" quantitatively, which implies that the generalization capability of diffusion models is adversely affected by the distance between high-density regions of target distributions.
- The theoretical findings are numerically verified in simulations.
The rest of this paper is organized as follows. In Section 2, we discuss the related work on the convergence and training fronts of diffusion models and also the generalization aspects of other generative modeling methods. Section 3 is the central part, which includes the problem formulation, main results, and consequences. Section 4 includes numerical verifications on synthetic and real-world datasets<sup>3</sup>. All the details of proofs and experiments are found in the appendices.
We review the related work on diffusion models concerning the central results in this paper.
- First, on the convergence theory, [7, 9] established elaborate error estimates between the modeled and target distribution given the discretization and time-dependent score matching
$^{3}$ Code is available at https://github.com/lphLeo/Diffusion_Generalization
Figure 1: Illustration of the problem formulation and important notations.
Target: Finitely-supported prob. & Gaussian mixtures Notations:
Loss: Time-dependent score matching (Eq. (7))
$t$  :SDE time  $T$  :maximal SDE time
Algorithm: Gradient flow
$p_{T}\approx \pi$  : a known prior
$\tau$  : training time
tolerance. Compared to the present work, they did not evolve the concrete training dynamics since the setting therein focuses on the properties of optimizers.
- Second, on the training front, [51] proposed a set of techniques to enhance the training performance of score-based generative models, scaling diffusion models to images of higher resolution, but without any characterization of possible generalization improvements. Similar to [7, 9], [49] also provided error estimates between the modeled and target distributions in a point-wise sense and again did not evolve the detailed training dynamics.
- Third, on the generalization and memorization side, corresponding theories are developed for bias potential models and GANs in [66] and [67], respectively, where the modeled distribution learns the ground truth with early-stopping and diverges or converges to the empirical distribution only associated with training samples after sufficiently long training time. The current work extends the mathematical analysis to the case of diffusion models under a data-dependent setting.
- As a supplement, we also discuss related literature regarding low-density learning. [50] illustrated the difficulty of learning from low-density regions with toy formulations and simulations, which motivates the sampling method of annealed Langevin dynamics as the predecessor of (score-based) diffusion models. [21] restudied similar problems under the pure score matching regime (without the denoising or time-dependent dynamics) and attributed the difficulty to increasing isoperimetry of distributions with modes shift. [41] defined the Hardness score and numerically justified that decreasing manifold densities leads to the increasing Hardness score, and consequently applied the Hardness score as the regularization to the sampling process to enhance synthetic images from low-density regions. As a comparison, this work establishes a mathematically rigorous estimate on the generalization gap that quantitatively depends on the range of low-density regions (between modes) in target distributions for (score-based) diffusion models that requires denoising.
3 Formulation and Results
In this section, we first introduce the problem setup. Next, we state the main theoretical results, subsequent consequences, and possible connections. The numerical illustration is provided at last.
Since DMs/SGMs have already grown into a large family of generative models with an enormous number of variants, there are various ways to define the parameterization of diffusion models. Here, we adopt (one of) the most fundamental architectures proposed in [54], where the forward perturbation and reverse sampling process are both implemented by a joint group of coupled (stochastic) differential equations. See Figure 1 for an illustration of the problem formulation.
Forward perturbation. We start with the setting of unsupervised learning. Given an unlabeled dataset  $\mathcal{D}_{\boldsymbol{x}} = \{\pmb{x}_i\}_{i=1}^n \subset \mathbb{R}^d$  with the sample  $\pmb{x}_i \stackrel{\text{i.i.d.}}{\sim} p_0(\pmb{x})$ , where  $p_0$  denotes the underlying (ground truth or target) distribution, the forward diffusion process is defined as
$$
d \boldsymbol {x} = \boldsymbol {f} (\boldsymbol {x}, t) d t + g (t) d \boldsymbol {W} _ {t}, \quad \boldsymbol {x} (0) \sim p _ {0}. \tag {1}
$$
Here, the drift coefficient  $\pmb{f}(\cdot ,t):\mathbb{R}^d\mapsto \mathbb{R}^d$  is a time-dependent vector-valued function, and the diffusion coefficient  $g(\cdot):\mathbb{R}_{\geq 0}\mapsto \mathbb{R}$  is a scalar function, and  $W_{t}$  denotes the standard Wiener process (a.k.a., Brownian motion). The SDE (1) has a unique, strong solution under certain regularity conditions (i.e., globally Lipschitz coefficients in both state and time; see [31]). From now on, we denote by  $p_t(\pmb {x}(t))$  the marginal distribution of  $\pmb {x}(t)$ , and let  $p_{t|s}(\pmb {x}(t)|\pmb {x}(s))$  be the (perturbation) transition kernel from  $\pmb {x}(s)$  to  $\pmb {x}(t)$ ,  $0\leq s < t\leq T < \infty$  with  $T$  as the time horizon. By appropriately selecting  $\pmb{f}$  and  $g$ , one can force the SDE (1) to converge to a prior distribution (typically a Gaussian). Common examples include the (time-rescaled) Ornstein-Uhlenbeck (OU) process, which is a special case of the linear version of (1):
$$
d \boldsymbol {x} = f (t) \boldsymbol {x} d t + g (t) d \boldsymbol {W} _ {t}, \quad \boldsymbol {x} (0) \sim p _ {0}. \tag {2}
$$
Reverse sampling. According to [1] and [54], both the following reverse-time SDE and probability flow ODE share the same marginal distribution as the forward-time SDE (1):
$$
d \boldsymbol {x} = \left[ \boldsymbol {f} (\boldsymbol {x}, t) - g ^ {2} (t) \nabla_ {\boldsymbol {x}} \log p _ {t} (\boldsymbol {x}) \right] d t + g (t) d \bar {\boldsymbol {W}} _ {t}, \tag {3}
$$
$$
d \boldsymbol {x} = \left[ \boldsymbol {f} (\boldsymbol {x}, t) - \frac {1}{2} g ^ {2} (t) \nabla_ {\boldsymbol {x}} \log p _ {t} (\boldsymbol {x}) \right] d t, \tag {4}
$$
where  $\bar{W}_t$  is a standard Wiener process when time flows backwards from  $T$  to 0, and  $dt$  is an infinitesimal negative time step. With the initial condition  $\pmb{x}(T) \sim p_T \approx \pi$ , where  $\pi$  is a known prior distribution such as the Gaussian noise, one can (numerically) solve (3) or (4) to transform noises into samples from  $p_0$ , which is exactly the goal of generative modeling.
Loss objectives. The only remaining task is to estimate the unknown (Stein) score function  $\nabla_{\pmb{x}}\log p_t(\pmb {x})$ . This is achieved by minimizing the following weighted sum of denoising score matching ([57]) objectives:
$$
\mathcal {L} (\boldsymbol {\theta}; \lambda (\cdot)) := \mathbb {E} _ {t \sim \mathcal {U} (0, T)} \left[ \lambda (t) \cdot \mathbb {E} _ {\boldsymbol {x} (0) \sim p _ {0}} \left[ \mathbb {E} _ {\boldsymbol {x} (t) \sim p _ {t | 0}} \left[ \| \boldsymbol {s} _ {t, \boldsymbol {\theta}} (\boldsymbol {x} (t)) - \nabla_ {\boldsymbol {x} (t)} \log p _ {t | 0} (\boldsymbol {x} (t) | \boldsymbol {x} (0)) \| _ {2} ^ {2} \right] \right] \right], \tag {5}
$$
with  $\pmb{\theta}^{*} := \arg \min_{\pmb{\theta}} \mathcal{L}(\pmb{\theta}; \lambda(\cdot))$ , where  $\mathcal{U}(0, T)$  denotes the uniform distribution over  $[0, T]$ , and  $\lambda(t): [0, T] \mapsto \mathbb{R}_{+}$  is a weighting function, which is typically selected as
$$
\lambda (t) \propto 1 / \sqrt {\mathbb {E} _ {\boldsymbol {x} (t) \sim p _ {t | 0}} [ \| \nabla_ {\boldsymbol {x} (t)} \log p _ {t | 0} (\boldsymbol {x} (t) | \boldsymbol {x} (0)) \| _ {2} ^ {2} ]} \tag {6}
$$
according to ([54]). The score function  $s_{t,\pmb{\theta}}: \mathbb{R}^d \mapsto \mathbb{R}^d$  is time-dependent and can be parameterized as a neural network (encoded with the time information) such as the U-net([37]) architecture commonly applied in the field of image segmentation. Alternatively, one can also define the time-dependent score matching loss
$$
\tilde {\mathcal {L}} (\boldsymbol {\theta}; \lambda (\cdot)) := \mathbb {E} _ {t \sim \mathcal {U} (0, T)} \left[ \lambda (t) \cdot \mathbb {E} _ {\boldsymbol {x} (t) \sim p _ {t}} \left[ \| \boldsymbol {s} _ {t, \boldsymbol {\theta}} (\boldsymbol {x} (t)) - \nabla_ {\boldsymbol {x} (t)} \log p _ {t} (\boldsymbol {x} (t)) \| _ {2} ^ {2} \right] \right], \tag {7}
$$
which is equivalent to (5) up to a constant independent of  $\theta$  by [57, 49].
In practice, expectations in the objective (5) can be respectively estimated with empirical means over time steps in  $[0,T]$ , data samples from  $p_0$  and  $p_{t|0}$ , which is efficient when the drift coefficient  $\pmb{f}(\cdot ,t)$  is linear. Specifically, if the forward-time SDE takes the form of (2), the transition kernel  $p_{t|0}$  has a closed form ([40])
$$
p _ {t \mid 0} (\boldsymbol {x} (t) | \boldsymbol {x} (0)) = \mathcal {N} (\boldsymbol {x} (t); r (t) \boldsymbol {x} (0), r ^ {2} (t) v ^ {2} (t) \boldsymbol {I} _ {d}), \tag {8}
$$
where  $\mathcal{N}(\pmb{x};\pmb{\mu},\pmb{\Sigma})$  denotes the multivariate Gaussian distribution evaluated at  $\pmb{x}$  with the expectation  $\pmb{\mu}$  and covariance  $\pmb{\Sigma}$ , and  $r(t)\coloneqq e^{\int_0^t f(\zeta)d\zeta}$ ,  $v(t)\coloneqq \sqrt{\int_0^t\frac{g^2(\zeta)}{r^2(\zeta)}d\zeta}$ .
4The OU process is the unique time-homogeneous Markov process which is also a Gaussian process, with the stationary distribution as the standard Gaussian distribution.
Training. We aim to investigate the gradient flow training dynamics over the empirical loss
$$
\frac {d}{d \tau} \hat {\boldsymbol {\theta}} _ {n} (\tau) = - \nabla_ {\hat {\boldsymbol {\theta}} _ {n} (\tau)} \hat {\mathcal {L}} _ {n} (\hat {\boldsymbol {\theta}} _ {n} (\tau); \lambda (\cdot)), \quad \hat {\boldsymbol {\theta}} _ {n} (0) := \hat {\boldsymbol {\theta}} _ {n} ^ {0}, \tag {9}
$$
where  $\hat{\mathcal{L}}_n$  is the Monte-Carlo estimation of  $\mathcal{L}$  defined in (5) on the training dataset, with an auxiliary gradient flow over the population loss
$$
\frac {d}{d \tau} \boldsymbol {\theta} (\tau) = - \nabla_ {\boldsymbol {\theta} (\tau)} \mathcal {L} (\boldsymbol {\theta} (\tau); \lambda (\cdot)), \quad \boldsymbol {\theta} (0) := \boldsymbol {\theta} ^ {0} = \hat {\boldsymbol {\theta}} _ {n} ^ {0}. \tag {10}
$$
In both cases, the weighting function  $\lambda(\cdot)$  is selected as in (6). Denote the score function learned at the training time  $\tau$  evaluated at the SDE time  $t$  with respect to the empirical loss and population loss as  $s_{t,\hat{\pmb{\theta}}_n(\tau)}(\pmb{x}(t))$  and  $s_{t,\pmb{\theta}(\tau)}(\pmb{x}(t))$ , respectively. The corresponding density functions, denoted by  $p_{t,\hat{\pmb{\theta}}_n(\tau)}(\pmb{x}(t))$  and  $p_{t,\pmb{\theta}(\tau)}(\pmb{x}(t))$ , are obtained by solving
$$
\nabla_ {\boldsymbol {x} (t)} \log p _ {t, \hat {\boldsymbol {\theta}} _ {n} (\tau)} (\boldsymbol {x} (t)) = \boldsymbol {s} _ {t, \hat {\boldsymbol {\theta}} _ {n} (\tau)} (\boldsymbol {x} (t)), \quad \nabla_ {\boldsymbol {x} (t)} \log p _ {t, \boldsymbol {\theta} (\tau)} (\boldsymbol {x} (t)) = \boldsymbol {s} _ {t, \boldsymbol {\theta} (\tau)} (\boldsymbol {x} (t)), \tag {11}
$$
and then normalizing, respectively.
Score networks. We parameterize the score function  $s_{t,\theta}$  as the following random feature model
$$
\boldsymbol {s} _ {t, \boldsymbol {\theta}} (\boldsymbol {x}) := \frac {1}{m} \boldsymbol {A} \sigma (\boldsymbol {W} \boldsymbol {x} + \boldsymbol {U} \boldsymbol {e} (t)) = \frac {1}{m} \sum_ {i = 1} ^ {m} \boldsymbol {a} _ {i} \sigma \left(\boldsymbol {w} _ {i} ^ {\top} \boldsymbol {x} + \boldsymbol {u} _ {i} ^ {\top} \boldsymbol {e} (t)\right), \tag {12}
$$
where  $\sigma$  is the ReLU activation function,  $\mathbf{A} = (\mathbf{a}_1,\dots ,\mathbf{a}_m)\in \mathbb{R}^{d\times m}$  is the trainable parameter, while  $\pmb {W} = (\pmb {w}_1,\dots ,\pmb {w}_m)^\top \in \mathbb{R}^{m\times d}$  and  $\pmb {U} = (\pmb {u}_1,\dots ,\pmb {u}_m)^\top \in \mathbb{R}^{m\times d_e}$  are randomly initialized and frozen during training, and  $e:\mathbb{R}_{\geq 0}\mapsto \mathbb{R}^{d_e}$  is the embedding function concerning the time information. Assume that  $\pmb {a}_i,\pmb {w}_i$  and  $\pmb {u}_i$  are i.i.d. sampled from an underlying distribution  $\rho$ . Then, as  $m\to \infty$ , we get
$$
\begin{array}{l} \boldsymbol {s} _ {t, \boldsymbol {\theta}} (\boldsymbol {x}) \rightarrow \bar {\boldsymbol {s}} _ {t, \bar {\boldsymbol {\theta}}} (\boldsymbol {x}) := \mathbb {E} _ {(\boldsymbol {a}, \boldsymbol {w}, \boldsymbol {u}) \sim \rho} [ \boldsymbol {a} \sigma (\boldsymbol {w} ^ {\top} \boldsymbol {x} + \boldsymbol {u} ^ {\top} \boldsymbol {e} (t)) ] \\ = \mathbb {E} _ {(\boldsymbol {w}, \boldsymbol {u}) \sim \rho_ {0}} \left[ \boldsymbol {a} (\boldsymbol {w}, \boldsymbol {u}) \sigma \left(\boldsymbol {w} ^ {\top} \boldsymbol {x} + \boldsymbol {u} ^ {\top} \boldsymbol {e} (t)\right) \right], \tag {13} \\ \end{array}
$$
with  $\pmb{a}(\pmb{w},\pmb{u}) := \frac{1}{\rho_0(\pmb{w},\pmb{u})}\int_{\mathbb{R}^d}\pmb{a}\rho(\pmb{a},\pmb{w},\pmb{u})d\pmb{a}$  and  $\rho_0(\pmb{w},\pmb{u}) := \int_{\mathbb{R}^d}\rho(\pmb{a},\pmb{w},\pmb{u})d\pmb{a}$ . By the positive homogeneity property of the ReLU activation, we can assume that  $\| \pmb{w}\|_1 + \| \pmb{u}\|_1 \leq 1$  w.l.o.g.
One can view  $\bar{s}_{t,\bar{\theta}}(\pmb{x})$  as a continuous version of the random feature model. Correspondingly, the optimal solution is denoted as  $\bar{\theta}^*$  when replacing the parameterized score function  $s_{t,\theta}(\pmb{x})$  in the loss objective (5) or (7) by  $\bar{s}_{t,\bar{\theta}}(\pmb{x})$ . Define the kernel  $k_{\rho_0}(\pmb{x},\pmb{x}') \coloneqq \mathbb{E}_{(\pmb{w},\pmb{u})\sim \rho_0}\left[\sigma (\pmb{w}^\top \pmb{x} + \pmb{u}^\top \pmb{e}(t))\sigma (\pmb{w}^\top \pmb{x}' + \pmb{u}^\top \pmb{e}(t))\right]$ , and let  $\mathcal{H}_{k_{\rho_0}}$  be the induced reproducing kernel Hilbert space (RKHS; [2]), we have  $\bar{s}_{t,\bar{\theta}} \in \mathcal{H}_{k_{\rho_0}}$  if the RKHS norm  $\| \bar{s}_{t,\bar{\theta}}\|_{\mathcal{H}_{k_{\rho_0}}}^2 \coloneqq \mathbb{E}_{(\pmb{w},\pmb{u})\sim \rho_0}[\| \pmb{a}(\pmb{w},\pmb{u})\|_2^2] = \| \| \pmb{a}\|_2\|_{L^2(\rho_0)}^2 < \infty$ , and the corresponding discrete version can be defined by the empirical average, i.e.,  $\| s_{t,\theta}\|_{\mathcal{H}_{k_{\rho_0}}}^2 \coloneqq \frac{1}{m}\| A\|_F^2 = \frac{1}{m}\sum_{i=1}^{m}\| \pmb{a}(\pmb{w}_i,\pmb{u}_i)\|_2^2$ .
Remark 1. There are more modern and complex mathematical tools such as neural tangent kernels (NTKs) and mean fields that can be selected as the score networks. Employing these modern tools is valuable at least for theoretical completeness and we leave these as the future work.<sup>6</sup>
The goal is to measure and bound the generalization error evolving with the gradient flow training dynamics (9) between the learned distribution and target distribution, using the common Kullback-Leibler (KL) divergence.
Definition 1 (KL divergence). Given two distributions  $p$  and  $q$ , the KL divergence from  $q$  to  $p$  is defined as  $D_{\mathrm{KL}}(p||q) = \int_{\mathbb{R}^d} p(\boldsymbol{x}) \log \left( \frac{p(\boldsymbol{x})}{q(\boldsymbol{x})} \right) d\boldsymbol{x}$ .
Based on the above definitions, the generalization gap along the gradient flow training dynamics (9) is formulated as  $D_{\mathrm{KL}}\left(p_0\| p_{0,\hat{\theta}_n(\tau)}\right)$ , which is only a function of the training time  $\tau$ . The goal is to estimate  $D_{\mathrm{KL}}\left(p_0\| p_{0,\hat{\theta}_n(\tau)}\right)$ .
Recall the dataset  $\mathcal{D}_{\boldsymbol{x}} = \{\boldsymbol{x}_i\}_{i=1}^n \subset \mathbb{R}^d$ , and we take  $\mathcal{D}_t = \{t_i\}_{i=1}^n \subset [0, T]$  for convenience.
For example, the extension to NTKs is possible, since the training of random feature models follows a specific NTK regime (only the last layer is updated) in the output space (instead of the parameter space), which can be properly analyzed in the infinite-width regime.
In this section, we state the main results of the generalization capability of the (score-based) diffusion models and how it evolves as the training proceeds. Based on the formulation in Section 3.1, we theoretically derive several upper bounds to estimate  $D_{\mathrm{KL}} \left(p_0 \| p_{0,\hat{\theta}_n(\tau)}\right)$  under different settings. The results cover both the positive and negative aspects: in the data-independent setting where the target distribution has finite support, the generalization gap is proved to be small; while in the data-dependent setting where the target distribution possesses shift modes, the generalization is adversely affected by the modes' distance.
3.2.1 Data-Independent Generalization Gap
In this section, we provide the characterization of the generalization capability for diffusion models given a target distribution defined on a finite domain. Generally, the KL divergence from the learned distribution  $p_{0,\hat{\theta}_n(\tau)}$  at the training time  $\tau$  to the target distribution  $p_0$  can be estimated as follows.
Theorem 1. Suppose that the target distribution  $p_0$  is continuously differentiable and has a compact support set, i.e.,  $||\pmb{x}||_{\infty}$  is uniformly bounded, and there exists a reproducing kernel Hilbert space (RKHS)  $\mathcal{H} \coloneqq \mathcal{H}_{k_{\rho_0}}$  such that  $\bar{s}_{0,\bar{\theta}^*} \in \mathcal{H}$ . Assume that the initial loss, trainable parameters, the embedding function  $e(t)$  and weighting function  $\lambda(t)$  are all bounded. Then for any  $\delta > 0$ ,  $\delta \ll 1$ , with the probability of at least  $1 - \delta$ , we have