SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Anant Raj
Coordinated Science Laboraotry University of Illinois Urbana-Champaign. Inria, Ecole Normale Supérieure PSL Research University, Paris, France. anant raj@inria.fr
Umut Şimsekli
Inria, CNRS, Ecole Normale Supérieure PSL Research University, Paris, France. umut.simsekli@inria.fr
Alessandro Rudi
Inria, Ecole Normale Supérieure PSL Research University, Paris, France. alessandro.rudi@inria.fr
This paper deals with the problem of efficient sampling from a stochastic differential equation, given the drift function and the diffusion matrix. The proposed approach leverages a recent model for probabilities [RC21] (the positive semi-definite - PSD model) from which it is possible to obtain independent and identically distributed (i.i.d.) samples at precision  $\varepsilon$  with a cost that is  $m^2 d\log(1/\varepsilon)$  where  $m$  is the dimension of the model,  $d$  the dimension of the space. The proposed approach consists of: first, computing the PSD model that satisfies the Fokker-Planck equation (or its fractional variant) associated with the SDE, up to error  $\varepsilon$ , and then sampling from the resulting PSD model. Assuming some regularity of the Fokker-Planck solution (i.e.  $\beta$ -times differentiability plus some geometric condition on its zeros) We obtain an algorithm that: (a) in the preparatory phase obtains a PSD model with L2 distance  $\varepsilon$  from the solution of the equation, with a model of dimension  $m = \varepsilon^{-(d+1)/(\beta-2s)}(\log(1/\varepsilon))^{d+1}$  where  $1/2 \leq s \leq 1$  is the fractional power to the Laplacian, and total computational complexity of  $O(m^{3.5}\log(1/\varepsilon))$  and then (b) for Fokker-Planck equation, it is able to produce i.i.d. samples with error  $\varepsilon$  in Wasserstein-1 distance, with a cost that is  $O(d\varepsilon^{-2(d+1)/\beta-2}\log(1/\varepsilon)^{2d+3})$  per sample. This means that, if the probability associated with the SDE is somewhat regular, i.e.  $\beta \geq 4d + 2$ , then the algorithm requires  $O(\varepsilon^{-0.88}\log(1/\varepsilon)^{4.5d})$  in the preparatory phase, and  $O(\varepsilon^{-1/2}\log(1/\varepsilon)^{2d+2})$  for each sample. Our results suggest that as the true solution gets smoother, we can circumvent the curse of dimensionality without requiring any sort of convexity.
High dimensional stochastic differential equations (SDE) and their associated partial differential equations (PDEs) arise often in various scientific and engineering applications such as control systems, aviation, fluid dynamics, etc [LT00, HF08, Trö10]. Sampling from an SDE is an active area of research. Generally, a direct approach like Langevin sampling [DMM19, DK19, CDJB20] is commonly employed to sample from SDEs. However, an alternative approach involves approximating the solution of the associated PDE in a form that enables easy sampling. In this paper, we focus on the latter approach.
The exact solutions of most of the high-dimensional PDEs are not computable in closed form. Hence, for this reason, finding approximate numerical solutions to high-dimensional SDEs and related PDEs has been an important area of research in numerical methods [Ame14, FGN13, Tad12]. With the advent of new machine learning methods, the question naturally arises as to whether we can leverage the expressive power of these models to effectively capture the solution of high-dimensional partial differential equations (PDEs). Recent research efforts have explored this direction by employing various machine learning models for different classes of PDEs [BHJK20, LLMD18, KKL $^{+}$ 21]. For instance, works such as [CHOS21, Ste21, BQL $^{+}$ 22] have harnessed the capabilities of reproducing kernel Hilbert spaces (RKHS) to model PDE solutions. With the recent developments in neural network models and architecture, [LLMD18, KLY21, KPRS22, CMW $^{+}$ 21] consider neural networks to model and approximate the solution of high dimensional PDEs. Despite impressive algorithmic innovation, the theoretical understanding of these approaches is still limited, leaving a significant class of PDEs beyond the reach of existing machine learning-based modeling frameworks or with inadequate theoretical insights into the obtained solutions.
In this paper, our goal is to sample from the stochastic differential equations (SDE) that are driven by a Brownian motion or an  $\alpha$ -stable process. The time evolution of probability density functions (PDFs) of stochastic differential equations (SDE) that are driven by a Brownian motion or an  $\alpha$ -stable process is given by Fokker-Planck Equation (FPE) and the fractional Fokker-Planck Equation (fractional FPE) respectively (see e.g., [DMM19, Sim17, HMW21]). Therefore, our focus in this study lies in approximating the solution of the Fokker-Planck Equation (FPE), along with its fractional counterpart, as discussed in [UHK18], and subsequently sampling from the approximated solution. The modeling of solutions for FPEs (FPEs) and fractional FPEs poses significant challenges, encompassing the following key aspects: (i) Solutions of FPEs and fractional FPEs are probability densities that are always non-negative and vanish at infinity, (ii) Modeling a probability density is hard because of the normalization property, and (iii) Fractional FPEs involve a fractional Laplacian operator that is non-local in nature. In order for an approximate solution of FPEs or fractional FPEs to be useful for sampling from the corresponding stochastic differential equations (SDEs), it is necessary that sampling from the approximate solution of the corresponding partial differential equation (PDE) be straightforward. Unfortunately, a comprehensive algorithmic approach that addresses the above-mentioned issues and includes a formal approximation analysis is currently lacking in the literature.
This paper adopts a positive semi-definite (PSD) model-based approach to effectively tackle the problem at hand. The recently proposed PSD model [MFBR20, RMFB20, RC21] offers promising solutions to the challenges encountered in probability density modeling, making it an excellent choice for representing the solution of FPEs and their fractional counterparts. In a recent work by [RC21], it was demonstrated that employing a Gaussian kernel in the PSD model allows for the exact determination of the normalization constant and enables the characterization of a broad range of probability densities using such PSD models. Moreover, an algorithm for sampling from the PSD model is also proposed recently with statistical guarantees [MFBR22].
Driven by these insights, we explore the utilization of a PSD model for approximating the solution of FPEs and fractional FPEs. Consequently, we employ the sampling algorithm presented in[MFBR22] to generate samples from the approximate solution. Our primary objective in this study is to obtain rigorous guarantees on the approximation error of PSD models when applied to the approximation of solutions of FPEs and fractional FPEs. Under regularity conditions on the solution of these PDEs, we make the following contributions.
Contributions: We make the following contributions in this paper:
- We show that the PSD-based representation of the solution of FPE has good approximation properties under the regularity assumptions on the true solution of the PDE. In particular, we show that we can achieve an approximation error of  $O(\varepsilon)$  while using  $m = O(\varepsilon^{-(d + 1) / (\beta - 2)}(\log \frac{1}{\varepsilon})^{(d + 1) / 2})$  number of samples for PSD representation where  $d$  is the dimension of space variables and the true solution is  $\beta$  times differentiable (Theorem 1 and 2). In addition, we are able to provide Wasserstein guarantees for the sampling error of a PSD model-driven solution when sampling from an SDE driven by a brownian motion (Corollary 1). An important contribution of our approach is that it relies on much weaker assumptions on the solution of associated PDE ( $\beta$  times differentiable) compared to existing methods that employ dissipativity, log-Sobolev, and Poincaré like conditions to provide sampling error guarantees [DMM19, DK19, Dal17, DM17,  $\mathrm{CEL}^{+}21$ ,  $\mathrm{BCE}^{+}22$ ].
- We demonstrate that utilizing a positive semi-definite (PSD) based representation is a natural and effective method to approximate the fractional Laplacian operator when applied to a probability density. In particular, we provide results that illustrate the effectiveness of PSD-based representations for Gaussian and shift-invariant kernels, which can be represented using Bochner's theorem (Theorem 3).
- As a final contribution, we also establish an approximation guarantee for the solution of the fractional FPE under certain regularity assumptions on the true solution. This extends our earlier result for the FPE and demonstrates the effectiveness of the PSD-based approach in approximating the solution of both FPE and fractional FPE. We achieve an approximation error of  $O(\varepsilon)$  while using  $m = O(\varepsilon^{-(d + 1) / (\beta -2s)}(\log \frac{1}{\varepsilon})^{(d + 1) / 2})$  number of samples for PSD representation where  $d$  is the dimension of space variables and the true solution is  $\beta$  times differentiable (Theorem 4 and 5). This result directly implies a Wasserstein guarantee on the sampling error when sampling from an SDE driven by an  $\alpha$ -stable levy process using the PSD model. To the best of our knowledge, our algorithm stands as the first of its kind, providing a provable guarantee for sampling from an SDE driven by an  $\alpha$ -stable levy process as well as for approximating the solution of fractional FPEs under regularity assumption on the solution.
Our results clearly suggest that as the true solution gets smoother, we require less number of samples to approximate the true solution using PSD-based representation, hence circumventing the curse of dimensionality. All the proofs are provided in the appendix.
[Eva22] provides a good introduction on PDEs. [GW79, Ise09, Ame14] have discussed numerical methods to solve a PDE at length.
Machine learning-based solution to PDEs: Modeling the solution of a PDE using machine learning approaches and in particular kernel methods is a well-established technique leading to optimal sample complexity and very efficient algorithms (see for example  $\mathrm{[BKO^{+}96}$ , SW06, Sch07, Fas07, PR20, CHOS21, Ste21]). With the recent advancement of neural networks in machine learning, new approaches have emerged. A deep learning-based approach that can handle general high-dimensional parabolic PDEs was introduced in [HJE18]. Two popular approaches that were proposed in this context are (i) Physics inspired neural networks (PINN) [RPK19] and (ii) Deep Ritz method  $\left[\mathrm{Y}^{+}18\right]$ . However, no statistical optimality was provided in the original papers. Recently,  $\mathrm{[LCL^{+}21]}$  studied the statistical limits of deep learning and Fourier basis techniques for solving elliptic PDEs. The sample complexity to obtain a solution via a PINN and Deep Ritz model are shown to be  $O\left(n^{-\frac{2\beta - 2}{d + 2\beta - 4}}\log n\right)$  and  $O\left(n^{-\frac{2\beta - 2}{d + 2\beta - 2}}\log n\right)$  respectively where  $\beta$  is the order of smoothness of the solution. Subsequently, in a series of papers [DRM22, LL22, LLW21, CLL21], the theoretical properties of solving a different class of PDEs are discussed under restrictive assumptions on the function class (barron space). For an overview, we refer to [BHJK20]. Solutions of PDEs that involve a fractional operator have also been modeled by function in RKHS [Arq18, AA20, ASA19, AAS20]. However, these works do not provide finite sample approximating error guarantees for the solution obtained. A review of kernel-based numerical methods for PDEs can be found in  $\mathrm{[SS^{+}16}$ , FF15].
The (fractional) Fokker-Planck equation: Various efforts have been made to approximate the solution of FPE and its fractional counterpart by numerical methods [TSIG22, PMB13, Yan13, LAT04, Bar01]. [KRP12, DM14, SS22] utilized a radial basis kernel-based approach to numerically compute the solution of an FPE. RKHS-based methods have also been used to obtain an approximate solution [BH17, SAA20]. However, the above mentioned works are numerical in nature and do not discuss the approximation limits of the solution. On the more applied side, various works have been proposed by harnessing the expressive power of deep networks to model the solution of FPEs and fractional FPEs  $\left[\mathrm{XZL}^{+}20\right.$ , ZDL22, WWLZ22]. However, a systematic theoretical study is missing from the literature.
Langevin Monte Carlo: An important line of research considers the probabilistic counterpart of FPE that is directly simulating the SDEs associated with the FPEs by time-discretization. Under convexity assumptions [Dal17, DM17, DM19] proved guarantees for obtaining a single (approximate) sample from the SDE for a given time. These frameworks were later extended in [RRT17, XCZG18, EH21], so that the convexity assumptions were weakened to more general notions such as uniform dissipativity. [MCF15] introduced a more general SDE framework, which covers a larger class of PDEs. In the case of sampling SDEs with domain constraints [BEL15, BDMP17, HKRC18, SR20]
proposed Langevin-based methods under convexity assumption, which were later weakened in [Lam21, ZL22]. [Sim17, NSR19] considered SDEs which corresponded to the fractional FPE in dimension one and this approach was later extended in [HMW21, ZZ22]. An overview of these methods is provided in [NF21]. Contrary to this line of research, which typically requires convexity or dissipativity, our framework solely requires smoothness conditions.
PSD Models: PSD models are an effective way to model non-negative functions and enjoy the nice properties of linear models and more generally RKHS. It was introduced in [MFBR20] and their effectiveness for modeling probability distributions was shown in [RC21]. It has been also effectively used in optimal transport estimation  $\mathrm{MVB}^{+}21$ ,  $\mathrm{VMR}^{+}21$ , in finding global minima of a non-convex function [RMFB20], in optimal control [BCRB22]. In [MFBR22], an algorithm to draw samples from a PSD model was proposed.
3 Notations and Background
Notation: Space input domain is denoted as  $\mathfrak{X}$  and time input domain is denoted as  $\mathfrak{T}$ . Combined space  $\mathcal{X} \times \mathcal{T}$  is denoted as  $\tilde{\mathcal{X}}$ .  $L^{\infty}(\tilde{\mathcal{X}})$ ,  $L^{1}(\tilde{\mathcal{X}})$  and  $L^{2}(\tilde{\mathcal{X}})$  denote respectively the space of essentially bounded, absolutely integrable and square-integrable functions with respect to Lebesgue measure over  $\tilde{\mathcal{X}}$ .  $W_{2}^{\beta}(\tilde{\mathcal{X}})$  denotes the Sobolev space of functions whose weak derivatives up to order  $\beta$  are square-integrable on  $\tilde{\mathcal{X}}$ . We denote by  $\mathbb{R}_{++}^{d}$  the space vectors in  $\mathbb{R}^d$  with positive entries,  $\mathbb{R}^{n \times d}$  the space of  $n \times d$  matrices,  $\mathbb{S}_+^n = \mathbb{S}_+(\mathbb{R}^n)$  the space of positive semi-definite  $n \times n$  matrices. Given a vector  $\eta \in \mathbb{R}^d$ , we denote  $\mathrm{diag}(\eta) \in \mathbb{R}^{d \times d}$  the diagonal matrix associated to  $\eta$ .
3.1 (Fractional) Fokker-Planck Equation
The FPE describes the time evaluation of particle density if the particles are moving in a vector field and are also influenced by random noise like Brownian motion. Let us consider the following SDE driven by the standard Brownian motion (Wiener process)  $W_{t}$ :
$$
d X _ {t} = \mu \left(X _ {t}, t\right) d t + \sigma d W _ {t} \tag {1}
$$
where  $X_{t}\in \mathbb{R}^{d}$  is a random variable,  $\mu (X_{t},t)\in \mathbb{R}^{d}$  is drift,  $\sigma \in \mathbb{R}^{d\times k}$  and  $W_{t}$  is a  $k$ -dimensional Wiener process.  $\mu_i(X_t,t)$  denotes the  $i$ th element of vector  $\mu (X_t,t)$ . The diffusion tensor $^1$  is defined by  $D = \frac{1}{2}\sigma \sigma^{\top}$ .
The probability density function  $p(x, t)$  of the random variable  $X_{t}$  solving the SDE (1) is given by the FPE, given as follows [RR96, Pav14]:
$$
\frac {\partial p (x , t)}{\partial t} = - \sum_ {i = 1} ^ {d} \frac {\partial}{\partial x _ {i}} \left(\mu_ {i} (x, t) p (x, t)\right) + \sum_ {i = 1} ^ {d} \sum_ {j = 1} ^ {d} D _ {i j} \frac {\partial^ {2}}{\partial x _ {i} \partial x _ {j}} p (x, t), \text {s u c h} p (0, x) = p _ {0} (x). \tag {2}
$$
It is known that when  $\mu(X_{t}, t)$  is the gradient of a potential or objective function  $f: \mathcal{X} \to \mathbb{R}$ , i.e.,  $\mu = \nabla f$ , where  $f$  satisfies certain smoothness and growth conditions, the stationary distribution of the stochastic process in equation (1) exists and is described by the so-called Gibbs distribution [Pav14].
Despite of having nice theoretical properties, Brownian-driven SDEs are not suitable in certain models in engineering, physics and biology [STL96, Dua15]. A popular non-Gaussian SDE used in such applications is given as follows:
$$
d X _ {t} = \mu \left(X _ {t}, t\right) d t + d L _ {t} ^ {\alpha}, \tag {3}
$$
where  $L_{t}^{\alpha}$  denotes the (rotationally invariant)  $\alpha$ -stable Lévy process in  $\mathbb{R}^{d}$  and is defined as follows for  $\alpha \in (0,2]$  [Dua15]:
1.  $L_0^\alpha = 0$  almost surely;
2. For any  $t_0 < t_1 < \dots < t_N$ , the increments  $L_{t_n}^\alpha - L_{t_{n-1}}^\alpha$  are independent;
3. The difference  $L_{t}^{\alpha} - L_{s}^{\alpha}$  and  $L_{t - s}^{\alpha}$  have the same distribution, with the characteristic function  $\mathbb{E}\left[e^{j\langle u,L_{t - s}^{\alpha}\rangle}\right] = \exp (-(t - s)^{\alpha}\| u\|_{2}^{\alpha})$  for  $t > s$  and  $j = \sqrt{-1}$ ;
4.  $L_{t}^{\alpha}$  is continuous in probability.
In the SDE described in equation (3), we choose  $\sigma = 1$  for the simplicity of the expressions. When  $\alpha = 2$ , the process recovers the Brownian motion, i.e.,  $L_{t}^{\alpha} = \sqrt{2} W_{t}$ . The fundamental difference
To simplify our analysis, we will consider diffusion tensor  $D$  independent from  $X_{t}$  and  $t$  in this work.
between  $L_{t}^{\alpha}$  and  $W_{t}$  when  $\alpha < 2$  is that the increments of this process are heavy-tailed, with the following property:  $\mathbb{E}[||L_t^\alpha ||^p ] = +\infty$  for all  $p\geq \alpha$  and  $t\geq 0$ . This heavy-tailed structure makes the solution paths of (3) discontinuous:  $(X_{t})_{t\geq 0}$  can exhibit at most countable number of discontinuities. SDEs based on  $\alpha$ -stable processes have also received some attention in machine learning as well [SSG19, HMW21, NSGR19, SSDE20, ZFM+20]. The resulting governing equation for stable-driven SDEs is similar to the traditional Fokker-Planck equation except that the order  $\alpha$  of the highest derivative is fractional when  $\alpha < 2$ . The following generalization of the FPE is known in the literature as fractional FPE because of the presence of fractional Laplacian operator [UHK18]:
$$
\frac {\partial p (x , t)}{\partial t} = - \sum_ {i = 1} ^ {d} \frac {\partial}{\partial x _ {i}} \left(\mu_ {i} (x, t) p (x, t)\right) - (- \Delta) ^ {\alpha / 2} p (x, t), \text {s u c h} p (0, x) = p _ {0} (x), \tag {4}
$$
where  $(- \Delta)^s$  is the non-local Laplacian operator which is also known as the fractional Laplacian operator and is induced by the stable process. For  $0 < s < 1$ , the fractional Laplacian of order  $s$ ,  $(- \Delta)^s$  can be defined on functions  $f: \mathbb{R}^d \to \mathbb{R}$  as a singular integral as follows:  $(- \Delta)^s f(x) = c_{d,s} \int_{\mathbb{R}^d} \frac{f(x) - f(y)}{\|x - y\|_2^{d + 2s}} dy$ , where  $c_{d,s} = \frac{4^s \Gamma(d/2 + s)}{\pi^{d/2} |\Gamma(-s)|}$ . A more useful representation of the fractional Laplacian operator is given as a Fourier multiplier,  $\mathcal{F}[(-\Delta)^s f](u) = \|u\|^{2s} \mathcal{F}[f](u)$ , where  $\mathcal{F}[f](u) = \int_{\mathbb{R}^d} f(x) e^{j x^\top u} dx$ . We will utilize the Fourier multiplier-based representation of the fractional Laplacian operator to prove our approximation result for the fractional FPE.
3.2 PSD Models for Probability Representation
Consider a feature map representation  $\phi : \mathcal{X} \to \mathcal{H}$  from input space  $\mathcal{X}$  to a Hilbert space  $\mathcal{H}$ , and a linear operator  $M \in \mathbb{S}_+(\mathcal{H})$ , then PSD models are represented as in [MFBR20],
$$
f (x; M, \phi) = \phi (x) ^ {\top} M \phi (x). \tag {5}
$$
It is clear that  $f(x; M, \phi)$  is a non-negative function and hence PSD models offer a general way to parameterize non-negative functions. We consider  $\mathcal{H}$  to be the RKHS corresponding to the kernel  $k$  such that  $\phi(x) = k(x, \cdot)$ . In particular, we will consider the case where:  $\phi = \phi_{\eta}: \mathbb{R}^{d} \to \mathcal{H}_{\eta}$  is the feature map associated with the Gaussian kernel  $k_{\eta}(x, y) = \phi_{\eta}(x)^{\top} \phi_{\eta}(y) = e^{-\eta \| x - y\|^{2}}$  with  $\eta > 0$ . Let us assume that  $M$  lies in the span of feature maps corresponding to  $n$  data points,  $X = \{x_{1}, \dots, x_{n}\}$  then  $M = \sum_{i,j} A_{ij} \phi(x_{i}) \phi(x_{j})^{\top}$  for some  $A \in \mathbb{S}_{+}^{n}$ . Hence, a Gaussian PSD model can be defined as,  $f(x; A, X, \eta) = \sum_{ij} A_{ij} k_{\eta}(x_{i}, x) k_{\eta}(x_{j}, x)$ . Given two base point matrices  $X \in \mathbb{R}^{n \times d}$  and  $X' \in \mathbb{R}^{m \times d}$ , then  $K_{X,X',\eta}$  denotes the kernel matrix with entries  $(K_{X,X',\eta})_{ij} = k_{\eta}(x_{i}, x_{j}')$  where  $x_{i}$  and  $x_{j}'$  are the  $i$ -th and  $j$ -th rows of  $X$  and  $X'$  respectively. Gaussian PSD model has a lot of interesting properties and the details can be found in [RC21]. But, we would like to mention below an important aspect of PSD models which enable them to be used as an effective representation for probability density:
Sum Rule (Marginalization and Integration): The integral of a PSD model can be computed as,
$$
\int f (x; A, X, \eta) d x = c _ {2 \eta} \operatorname {T r} \left(A K _ {X, X, \frac {\eta}{2}}\right), \text {w h e r e} c _ {\eta} = \int k (0, x) d x.
$$
Similarly, only one variable of a PSD model can also be integrated to result in the sum rule.
Proposition 1 (Marginalization in one variable [RC21]). Let  $X \in \mathbb{R}^{n \times d}$ ,  $Y \in \mathbb{R}^{\times \times'}$ ,  $A \in \mathbb{S}_{+}(\mathbb{R}^{n})$  and  $\eta, \eta' \in \mathbb{R}+$ , then the following integral is a PSD model,
$$
\int f (x, y; A, [ X, Y ], (\eta , \eta^ {\prime})) d x = f (y; B, Y, \eta^ {\prime}) w i t h B = c _ {2 \eta} A \circ K _ {X, X, \frac {\eta}{2}}.
$$
In a recent work, [MFBR22] gave an efficient algorithm to sample from a PSD model, with the following result
Proposition 2 (Efficient sampling from PSD models [MFBR22]). Let  $\varepsilon >0$ . Given a PSD model  $f(x;A,\eta)$  with  $A\in \mathbb{R}^{m\times m}$  for  $x\in \mathbb{R}^d$ . There exists an algorithm (presented in [MFBR22]) that samples i.i.d. points from a probability  $\tilde{f}_{\varepsilon}$  such that the 1-Wasserstein distance between  $f(\cdot ;A,\eta)$  and  $\tilde{f}_{\varepsilon}$  satisfies  $\mathbb{W}_1(f,\tilde{f}_{\varepsilon})\leq \varepsilon$ . Moreover, the cost of the algorithm is  $O(m^{2}d\log (d / \varepsilon))$
4 Approximation of the (Fractional) Fokker-Planck Equation
In this section, we investigate the approximating properties of PSD models for solving FPEs and fractional FPEs. Our focus is on the approximation guarantee achieved by fitting a single PSD model
in the joint space-time domain. Let's denote the solution of the PDE as  $p^{\star}(x,t)$ , where we consider the FPE in section 4.1 and the fractional FPE in section 4.2. We introduce mild assumptions on the domain and the solution probability density. Throughout the paper, we assume the spatial domain  $\mathcal{X}$  to be  $(-R,R)^d$  and the time domain  $\mathcal{T}$  to be  $(0,R)$ . We have the following assumption from [RC21] on the solution  $p^{\star}$  of the PDE,