Title: Efficient Sampling of Stochastic Differential Equations with Positive Semi-Definite Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Kernel Methods

Sampling

Fokker-Planck Equation

Fractional Fokker-Planck Equation

Stochastic Differential Equations

Partial Differential Equations
Abstract: This paper deals with the problem of efficient sampling from a stochastic differential equation, given the drift function and the diffusion matrix. The proposed approach leverages a recent model for probabilities (Rudi and Ciliberto, 2021) (the positive semi-definite -- PSD model) from which it is possible to obtain independent and identically distributed (i.i.d.) samples at precision $\varepsilon$ with a cost that is $m^2 d \log(1/\varepsilon)$ where $m$ is the dimension of the model, $d$ the dimension of the space. The proposed approach consists in: first, computing the PSD model that satisfies the Fokker-Planck equation (or its fractional variant) associated with the SDE, up to error $\varepsilon$, and then sampling from the resulting PSD model. Assuming some regularity of the Fokker-Planck solution (i.e. $\beta$-times differentiability plus some geometric condition on its zeros) We obtain an algorithm that: (a) in the preparatory phase obtains a PSD model with L2 distance $\varepsilon$ from the solution of the equation, with a model of dimension $m = \varepsilon^{-(d+1)/(\beta-2s)} (\log(1/\varepsilon))^{d+1}$ where $1/2\leq s\leq1$ is the fractional power to the Laplacian, and total computational complexity of $O(m^{3.5} \log(1/\varepsilon))$ and then (b) for Fokker-Planck equation, it is able to produce i.i.d.\ samples with error $\varepsilon$ in Wasserstein-1 distance, with a cost that is $O(d \varepsilon^{-2(d+1)/\beta-2} \log(1/\varepsilon)^{2d+3})$ per sample. This means that, if the probability associated with the SDE is somewhat regular, i.e. $\beta \geq 4d+2$, then the algorithm requires $O(\varepsilon^{-0.88} \log(1/\varepsilon)^{4.5d})$ in the preparatory phase, and $O(\varepsilon^{-1/2}\log(1/\varepsilon)^{2d+2})$ for each sample. Our results suggest that as the true solution gets smoother, we can circumvent the curse of dimensionality without requiring any sort of convexity.

Efficient Sampling of Stochastic Differential Equations with Positive Semi-Definite Models

Abstract

This paper deals with the problem of efficient sampling from a stochastic differential equation, given the drift function and the diffusion matrix. The proposed approach leverages a recent model for probabilities [RC21] (the positive semi-definite - PSD model) from which it is possible to obtain independent and identically distributed (i.i.d.) samples at precision ε with a cost that is m 2 d log(1 /ε ) where m is the dimension of the model, d the dimension of the space. The proposed approach consists of: first, computing the PSD model that satisfies the Fokker-Planck equation (or its fractional variant) associated with the SDE, up to error ε , and then sampling from the resulting PSD model. Assuming some regularity of the FokkerPlanck solution (i.e. β -times differentiability plus some geometric condition on its zeros) We obtain an algorithm that: (a) in the preparatory phase obtains a PSD model with L2 distance ε from the solution of the equation, with a model of dimension m = ε -( d +1) / ( β -2 s ) (log(1 /ε )) d +1 where 1 / 2 ≤ s ≤ 1 is the fractional power to the Laplacian, and total computational complexity of O ( m 3 . 5 log(1 /ε )) and then (b) for Fokker-Planck equation, it is able to produce i.i.d. samples with error ε in Wasserstein-1 distance, with a cost that is O ( dε -2( d +1) /β -2 log(1 /ε ) 2 d +3 ) per sample. This means that, if the probability associated with the SDE is somewhat regular, i.e. β ≥ 4 d + 2 , then the algorithm requires O ( ε -0 . 88 log(1 /ε ) 4 . 5 d ) in the preparatory phase, and O ( ε -1 / 2 log(1 /ε ) 2 d +2 ) for each sample. Our results suggest that as the true solution gets smoother, we can circumvent the curse of dimensionality without requiring any sort of convexity.

1 Introduction

High dimensional stochastic differential equations (SDE) and their associated partial differential equations (PDEs) arise often in various scientific and engineering applications such as control systems, aviation, fluid dynamics, etc [LT00, HF08, Trö10]. Sampling from an SDE is an active area of research. Generally, a direct approach like Langevin sampling [DMM19, DK19, CDJB20] is commonly employed to sample from SDEs. However, an alternative approach involves approximating the solution of the associated PDE in a form that enables easy sampling. In this paper, we focus on the latter approach.

The exact solutions of most of the high-dimensional PDEs are not computable in closed form. Hence, for this reason, finding approximate numerical solutions to high-dimensional SDEs and related PDEs has been an important area of research in numerical methods [Ame14, FGN13, Tad12]. With the advent of new machine learning methods, the question naturally arises as to whether we can leverage the expressive power of these models to effectively capture the solution of high-dimensional partial differential equations (PDEs). Recent research efforts have explored this direction by employing various machine learning models for different classes of PDEs [BHJK20, LLMD18, KKL + 21]. For instance, works such as [CHOS21, Ste21, BQL + 22] have harnessed the capabilities of reproducing kernel Hilbert spaces (RKHS) to model PDE solutions. With the recent developments in neural network models and architecture, [LLMD18, KLY21, KPRS22, CMW + 21] consider neural networks to model and approximate the solution of high dimensional PDEs. Despite impressive algorithmic innovation, the theoretical understanding of these approaches is still limited, leaving a significant class of PDEs beyond the reach of existing machine learning-based modeling frameworks or with inadequate theoretical insights into the obtained solutions.

In this paper, our goal is to sample from the stochastic differential equations (SDE) that are driven by a Brownian motion or an α -stable process. The time evolution of probability density functions (PDFs) of stochastic differential equations (SDE) that are driven by a Brownian motion or an α -stable process is given by Fokker-Planck Equation (FPE) and the fractional Fokker-Planck Equation (fractional FPE) respectively (see e.g, [DMM19, ¸ Sim17, HMW21]. Therefore, our focus in this study lies in approximating the solution of the Fokker-Planck Equation (FPE), along with its fractional counterpart, as discussed in [UHK18], and subsequently sampling from the approximated solution. The modeling of solutions for FPEs (FPEs) and fractional FPEs poses significant challenges, encompassing the following key aspects: (i) Solutions of FPEs and fractional FPEs are probability densities that are always non-negative and vanish at infinity, (ii) Modeling a probability density is hard because of the normalization property, and (iii) Fractional FPEs involve a fractional Laplacian operator that is non-local in nature. In order for an approximate solution of FPEs or fractional FPEs to be useful for sampling from the corresponding stochastic differential equations (SDEs), it is necessary that sampling from the approximate solution of the corresponding partial differential equation (PDE) be straightforward. Unfortunately, a comprehensive algorithmic approach that addresses the abovementioned issues and includes a formal approximation analysis is currently lacking in the literature.

This paper adopts a positive semi-definite (PSD) model-based approach to effectively tackle the problem at hand. The recently proposed PSD model [MFBR20, RMFB20, RC21] offers promising solutions to the challenges encountered in probability density modeling, making it an excellent choice for representing the solution of FPEs and their fractional counterparts. In a recent work by [RC21], it was demonstrated that employing a Gaussian kernel in the PSD model allows for the exact determination of the normalization constant and enables the characterization of a broad range of probability densities using such PSD models. Moreover, an algorithm for sampling from the PSD model is also proposed recently with statistical guarantees [MFBR22].

Driven by these insights, we explore the utilization of a PSD model for approximating the solution of FPEs and fractional FPEs. Consequently, we employ the sampling algorithm presented in[MFBR22] to generate samples from the approximate solution. Our primary objective in this study is to obtain rigorous guarantees on the approximation error of PSD models when applied to the approximation of solutions of FPEs and fractional FPEs. Under regularity conditions on the solution of these PDEs, we make the following contributions.

Contributions: We make the following contributions in this paper:

We show that the PSD-based representation of the solution of FPE has good approximation properties under the regularity assumptions on the true solution of the PDE. In particular, we show that we can achieve an approximation error of O ( ε ) while using m = O ( ε -( d +1) / ( β -2) (log 1 ε ) ( d +1) / 2 ) number of samples for PSD representation where d is the dimension of space variables and the true solution is β times differentiable ( Theorem 1 and 2 ). In addition, we are able to provide Wasserstein guarantees for the sampling error of a PSD model-driven solution when sampling from an SDE driven by a brownian motion ( Corollary 1 ). An important contribution of our approach is that it relies on much weaker assumptions on the solution of associated PDE ( β times differentiable) compared to existing methods that employ dissipativity, log-Sobolev, and Poincaré like conditions to provide sampling error guarantees [DMM19, DK19, Dal17, DM17, CEL + 21, BCE + 22].

We demonstrate that utilizing a positive semi-definite (PSD) based representation is a natural and effective method to approximate the fractional Laplacian operator when applied to a probability density. In particular, we provide results that illustrate the effectiveness of PSD-based representations for Gaussian and shift-invariant kernels, which can be represented using Bochner's theorem ( Theorem 3 ).

As a final contribution, we also establish an approximation guarantee for the solution of the fractional FPE under certain regularity assumptions on the true solution. This extends our earlier result for the FPE and demonstrates the effectiveness of the PSD-based approach in approximating the solution of both FPE and fractional FPE We achieve an approximation error of O ( ε ) while using m = O ( ε -( d +1) / ( β -2 s ) (log 1 ε ) ( d +1) / 2 ) number of samples for PSD representation where d is the dimension of space variables and the true solution is β times differentiable ( Theorem 4 and 5 ). This result directly implies a Wasserstein guarantee on the sampling error when sampling from an SDE driven by an α -stable levy process using the PSD model. To the best of our knowledge, our algorithm stands as the first of its kind, providing a provable guarantee for sampling from an SDE driven by an α -stable levy process as well as for approximating the solution of fractional FPEs under regularity assumption on the solution.

Our results clearly suggest that as the true solution gets smoother, we require less number of samples to approximate the true solution using PSD-based representation, hence circumventing the curse of dimensionality. All the proofs are provided in the appendix.

2 Related Work

[Eva22] provides a good introduction on PDEs. [GW79, Ise09, Ame14] have discussed numerical methods to solve a PDE at length.

Machine learning-based solution to PDEs: Modeling the solution of a PDE using machine learning approaches and in particular kernel methods is a well-established technique leading to optimal sample complexity and very efficient algorithms (see for example [BKO + 96, SW06, Sch07, Fas07, PR20, CHOS21, Ste21]). With the recent advancement of neural networks in machine learning, new approaches have emerged. A deep learning-based approach that can handle general high-dimensional parabolic PDEs was introduced in [HJE18]. Two popular approaches that were proposed in this context are (i) Physics inspired neural networks (PINN) [RPK19] and (ii) Deep Ritz method [Y + 18]. However, no statistical optimality was provided in the original papers. Recently, [LCL + 21] studied the statistical limits of deep learning and Fourier basis techniques for solving elliptic PDEs. The sample complexity to obtain a solution via a PINN and Deep Ritz model are shown to be O ( n -2 β -2 d +2 β -4 log n ) and O ( n -2 β -2 d +2 β -2 log n ) respectively where β is the order of smoothness of the solution. Subsequently, in a series of papers [DRM22, LL22, LLW21, CLL21], the theoretical properties of solving a different class of PDEs are discussed under restrictive assumptions on the function class (barron space). For an overview, we refer to [BHJK20]. Solutions of PDEs that involve a fractional operator have also been modeled by function in RKHS [Arq18, AA20, ASA19, AAS20]. However, these works do not provide finite sample approximating error guarantees for the solution obtained. A review of kernel-based numerical methods for PDEs can be found in [SS + 16, FF15].

The (fractional) Fokker-Planck equation: Various efforts have been made to approximate the solution of FPE and its fractional counterpart by numerical methods [TSIG22, PMB13, Yan13, LAT04, Bar01]. [KRP12, DM14, SS22] utilized a radial basis kernel-based approach to numerically compute the solution of an FPE. RKHS-based methods have also been used to obtain an approximate solution [BH17, SAA20]. However, the above mentioned works are numerical in nature and do not discuss the approximation limits of the solution. On the more applied side, various works have been proposed by harnessing the expressive power of deep networks to model the solution of FPEs and fractional FPEs [XZL + 20, ZDL22, WWLZ22]. However, a systematic theoretical study is missing from the literature.

Langevin Monte Carlo: An important line of research considers the probabilistic counterpart of FPE that is directly simulating the SDEs associated with the FPEs by time-discretization. Under convexity assumptions [Dal17, DM17, DM19] proved guarantees for obtaining a single (approximate) sample from the SDE for a given time. These frameworks were later extended in [RRT17, XCZG18, EH21], so that the convexity assumptions were weakened to more general notions such as uniform dissipativity. [MCF15] introduced a more general SDE framework, which covers a larger class of PDEs. In the case of sampling SDEs with domain constraints [BEL15, BDMP17, HKRC18, SR20]

proposed Langevin-based methods under convexity assumption, which were later weakened in [Lam21, ZL22]. [¸ Sim17, NSR19] considered SDEs which corresponded to the fractional FPE in dimension one and this approach was later extended in [HMW21, ZZ22]. An overview of these methods is provided in [NF21]. Contrary to this line of research, which typically requires convexity or dissipativity, our framework solely requires smoothness conditions.

PSD Models: PSD models are an effective way to model non-negative functions and enjoy the nice properties of linear models and more generally RKHS. It was introduced in [MFBR20] and their effectiveness for modeling probability distributions was shown in [RC21]. It has been also effectively used in optimal transport estimation [MVB + 21, VMR + 21], in finding global minima of a non-convex function [RMFB20], in optimal control [BCRB22]. In [MFBR22], an algorithm to draw samples from a PSD model was proposed.

3 Notations and Background

Notation: Space input domain is denoted as X and time input domain is denoted as T . Combined space X × T is denoted as ˜ X . L ∞ ( ˜ X ) , L 1 ( ˜ X ) and L 2 ( ˜ X ) denote respectively the space of essentially bounded, absolutely integrable and square-integrable functions with respect to Lebesgue measure over ˜ X . W β 2 ( ˜ X ) denotes the Sobolev space of functions whose weak derivatives up to order β are square-integrable on ˜ X . We denote by R d ++ the space vectors in R d with positive entries, R n × d the space of n × d matrices, S n + = S + ( R n ) the space of positive semi-definite n × n matrices. Given a vector η ∈ R d , we denote diag( η ) ∈ R d × d the diagonal matrix associated to η .

3.1 (Fractional) Fokker-Planck Equation

The FPE describes the time evaluation of particle density if the particles are moving in a vector field and are also influenced by random noise like Brownian motion. Let us consider the following SDE driven by the standard Brownian motion (Wiener process) W t :

where X t ∈ R d is a random variable, µ ( X t , t ) ∈ R d is drift, σ ∈ R d × k and W t is a k -dimensional Wienner process. µ i ( X t , t ) denotes the i th element of vector µ ( X t , t ) . The diffusion tensor 1 is defined by D = 1 2 σσ ⊤ .

The probability density function p ( x, t ) of the random variable X t solving the SDE (1) is given by the FPE, given as follows [RR96, Pav14]:

It is known that when µ ( X t , t ) is the gradient of a potential or objective function f : X → R , i.e., µ = ∇ f , where f satisfies certain smoothness and growth conditions, the stationary distribution of the stochastic process in equation (1) exists and is described by the so-called Gibbs distribution [Pav14].

Despite of having nice theoretical properties, Brownian-driven SDEs are not suitable in certain models in engineering, physics and biology [STL96, Dua15]. A popular non-Gaussian SDE used in such applications is given as follows:

where L α t denotes the (rotationally invariant) α -stable Lévy process in R d and is defined as follows for α ∈ (0 , 2] [Dua15]:

L α 0 = 0 almost surely;

For any t 0 < t 1 < · · · < t N , the increments L α t n -L α t n -1 are independent;

The difference L α t -L α s and L α t -s have the same distribution, with the characteristic function E [ e j ⟨ u,L α t -s ⟩ ] = exp( -( t -s ) α ∥ u ∥ α 2 ) for t > s and j = √ -1 ;

L α t is continuous in probability.

In the SDE described in equation (3), we choose σ = 1 for the simplicity of the expressions. When α = 2 , the process recovers the Brownian motion, i.e., L α t = √ 2 W t . The fundamental difference

between L α t and W t when α < 2 is that the increments of this process are heavy-tailed, with the following property: E [ ∥ L α t ∥ p ] = + ∞ for all p ≥ α and t ≥ 0 . This heavy-tailed structure makes the solution paths of (3) discontinuous: ( X t ) t ≥ 0 can exhibit at most countable number of discontinuities. SDEs based on α -stable processes have also received some attention in machine learning as well [SSG19, HMW21, NSGR19, SSDE20, ZFM + 20]. The resulting governing equation for stable-driven SDEs is similar to the traditional Fokker-Planck equation except that the order α of the highest derivative is fractional when α < 2 . The following generalization of the FPE is known in the literature as fractional FPE because of the presence of fractional Laplacian operator [UHK18]:

where ( -∆) s is the non-local Laplacian operator which is also known as the fractional Laplacian operator and is induced by the stable process. For 0 < s < 1 , the fractional Laplacian of order s , ( -∆) s can be defined on functions f : R d → R as a singular integral as follows: ( -∆) s f ( x ) = c d,s ∫ R d f ( x ) -f ( y ) ∥ x -y ∥ d +2 s 2 dy, where c d,s = 4 s Γ( d/ 2+ s ) π d/ 2 | Γ( -s ) | . A more useful representation of the fractional Laplacian operator is given as a Fourier multiplier, F [( -∆) s f ]( u ) = ∥ u ∥ 2 s F [ f ]( u ) , where F [ f ]( u ) = ∫ R d f ( x ) e jx ⊤ u dx . We will utilize the Fourier multiplier-based representation of the fractional Laplacian operator to prove our approximation result for the fractional FPE.

3.2 PSD Models for Probability Representation

Consider a feature map representation ϕ : X → H from input space X to a Hilbert space H , and a linear operator M ∈ S + ( H ) , then PSD models are represented as in [MFBR20],

It is clear that f ( x ; M,ϕ ) is a non-negative function and hence PSD models offer a general way to parameterize non-negative functions . We consider H to be the RKHS corresponding to the kernel k such that ϕ ( x ) = k ( x, · ) . In particular, we will consider the case where: ϕ = ϕ η : R d → H η is the feature map associated with the Gaussian kernel k η ( x, y ) = ϕ η ( x ) ⊤ ϕ η ( y ) = e -η ∥ x -y ∥ 2 with η > 0 . Let us assume that M lies in the span of feature maps corresponding to n data points, X = { x 1 , · · · , x n } then M = ∑ i,j A ij ϕ ( x i ) ϕ ( x j ) ⊤ for some A ∈ S n + . Hence, a Gaussian PSD model can be defined as, f ( x ; A,X,η ) = ∑ ij A ij k η ( x i , x ) k η ( x j , x ) . Given two base point matrices X ∈ R n × d and X ′ ∈ R m × d , then K X,X ′ ,η denotes the kernel matrix with entries ( K X,X ′ ,η ) ij = k η ( x i , x ′ j ) where x i and x ′ j are the i -th and j -th rows of X and X ′ respectively. Gaussian PSD model has a lot of interesting properties and the details can be found in [RC21]. But, we would like to mention below an important aspect of PSD models which enable them to be used as an effective representation for probability density:

Sum Rule (Marginalization and Integration) : The integral of a PSD model can be computed as,

Similarly, only one variable of a PSD model can also be integrated to result in the sum rule.

Proposition 1 (Marginalization in one variable [RC21]) . Let X ∈ R n × d , Y ∈ R ⋉ × ′ , A ∈ S + ( R n ) and η, η ′ ∈ R + , then the following integral is a PSD model,

In a recent work, [MFBR22] gave an efficient algorithm to sample from a PSD model, with the following result

Proposition 2 (Efficient sampling from PSD models [MFBR22]) . Let ε > 0 . Given a PSD model f ( x ; A,η ) with A ∈ R m × m for x ∈ R d . There exists an algorithm (presented in [MFBR22]) that samples i.i.d. points from a probability ˜ f ε such that the 1-Wasserstein distance between f ( · ; A,η ) and ˜ f ε satisfies W 1 ( f, ˜ f ε ) ≤ ε. Moreover, the cost of the algorithm is O ( m 2 d log( d/ε )) .

4 Approximation of the (Fractional) Fokker-Planck Equation

In this section, we investigate the approximating properties of PSD models for solving FPEs and fractional FPEs. Our focus is on the approximation guarantee achieved by fitting a single PSD model

in the joint space-time domain. Let's denote the solution of the PDE as p ⋆ ( x, t ) , where we consider the FPE in section 4.1 and the fractional FPE in section 4.2. We introduce mild assumptions on the domain and the solution probability density. Throughout the paper, we assume the spatial domain X to be ( -R,R ) d and the time domain T to be (0 , R ) . We have the following assumption from [RC21] on the solution p ⋆ of the PDE,

Assumption 1. Let β > 2 , q ∈ N . There exists f 1 , f 2 , · · · , f q ∈ W β 2 ( ˜ X ) ∩ L ∞ ( ˜ X ) , such that the density p ⋆ : ˜ X → R satisfies, p ⋆ ( x, t ) = ∑ q i = j f 2 j ( x, t ) .

The assumption above is quite general and satisfied by a wide family of probabilities, as discussed in the Proposition 5 of [RC21]. We reiterate the result here below.

Proposition 3 (Proposition 5, [RC21]) . The assumption above is satisfied by

(i) any p ⋆ ( x, t ) that is β -times differentiable and strictly positive on [ -R,R ] d × [0 , R ] ,

(ii) any exponential model p ( x, t ) = e -v ( x,t ) such that v ∈ W β 2 ( ˜ X ) ∩ L ∞ ( ˜ X ) ,

(iii) any mixture model of Gaussians or, more generally, of exponential models from (ii),

(iv) any p that is β +2 -times differentiable on [ -R,R ] d × [0 , R ] with a finite set of zeros in ˜ X and with positive definite Hessian in each zero.

Moreover when p ⋆ is β -times differentiable over [ -R,R ] d × [0 , R ] , then it belongs to W β 2 ( ˜ X ) ∩ L ∞ ( ˜ X ) .

We will need to impose some extra mild assumptions on f j for j ∈ { 1 , · · · , q } to approximate the solution of a fractional Fokker-Planck equation because of involvement of the fractional laplacian operator (non-local operator). We will cite these extra assumptions when we describe the approximation results in section 4.2. Apart from that, we also have a boundedness assumption for the coefficients of the PDE in the domain we consider.

Assumption 2. We have for all ( x, t ) ∈ ˜ X , µ i ( x, t ) ≤ R µ , ∂µ i ( x,t ) ∂x i ≤ R µ p , and D ij ≤ R d for all ( i, j ) ∈ { 1 , · · · , d } .

It is also obvious to see that the PSD-based representation of probability density vanishes at infinity.

4.1 Approximating Solution of Fokker-Planck Equation

Let us consider the Fokker-Planck equation given in equation (2) and p ⋆ ( x, t ) is the solution of Fokker-Planck equation in equation (2) which satisfies the assumption 1. Hence, p ⋆ ( x, t ) satisfies the following,

For now, we are ignoring the initial value as we only are interested in approximation error and we will have a uniform approximation of the solution density p ⋆ and its derivatives up to the order 2 in the domain.

Given m base points pair (˜ x i , ˜ t i ) ∈ ˜ X for i ∈ { 1 , 2 , · · · , m } of space and time, let us consider the following approximation model for the solution of FPE,

where k X,T is the joint kernel across space and time and A is a positive semi-definite matrix of dimension R m × m . k X,T can be defined as the product of two Gaussian kernels k X and k T which are defined on space and time domain respectively i.e k X,T (( x, t ) , ( x ′ , t ′ )) = k X ( x, x ′ ) · k T ( t, t ′ ) . We represent ϕ X ( x ) = k X ( x, · ) ∈ H X , ϕ T ( t ) = k T ( t, · ) ∈ H T and ϕ X,T ( x, t ) = ϕ X ( x ) ⊗ ϕ T ( t ) . Here, we assume that k X and k T are both Gaussian kernels and hence the RKHS H X and H T correspond to the RKHS of the Gaussian kernel. For simplicity, we keep the kernel bandwidth parameter η the same for k X and k T . It is clear that the joint kernel k X,T is also Gaussian with the kernel bandwidth parameter equal to η . Given M = ∑ i,j A ij ϕ X,T ( x i , t i ) ϕ X,T ( x j , t j ) ⊤ , p ( x, t ) can also be represented with function f (( x, t ) , A, X, η ) as,

where ψ ( x, t ) ∈ R n , [ ψ ( x, t )] i = k X ( x, ˜ x i ) · k T ( t, ˜ t i ) and ˜ X ∈ R m × d +1 is the concatenated data matrix of space and time for m pair of samples ( x i , t i ) for i ∈ { 1 , · · · , m } .

We can now easily optimize an associate loss function with the FPE to obtain the unknown PSD matrix A . Let us assume that there exists a positive semi-definite matrix A for which the PSD model ˜ p ( x, t ) approximates the true solution p ⋆ ( x, t ) well. In that case, our goal is to obtain an upper bound on the following error objective:

Our approximation guarantee works in two steps. In the first step, we find an infinite dimensional positive operator M ε : H X ⊗ H T → H X ⊗ H T such that if we denote ˆ p ( x, t ) = ϕ X,T ( x, t ) ⊤ M ε ϕ X,T ( x, t ) then for some M ε , ˆ p is ε -approximation of the true density p ⋆ . Clearly, for a set of functions ˜ f 1 , · · · , ˜ f q ∈ H X ⊗ H T , M ε = ∑ q j =1 ˜ f j ˜ f j ⊤ is a positive operator from H X ⊗ H T → H X ⊗ H T . Now, we carefully construct ˜ f j for j ∈ { 1 , · · · , q } in RKHS H X ⊗ H T such that approximation guarantee in theorem 1 is satisfied. Details of the construction is given in Appendix. We represent ( x, t ) as a d +1 dimensional vector ˜ x and a mollifier function g (details are in the appendix) such that g v ( x ) = v -( d +1) g (˜ x/v ) . Then, we construct our ˜ f j = f j ⋆ g v for j ∈ { 1 , 2 , · · · , q } . We prove the result in Theorem 1 with this construction. To prove our approximation error guarantee for the solution of FPE using PSD model, we assume the derivatives of f 1 , · · · , f q up to order 2 are bounded which is already covered in Assumption 1.

Theorem 1. Let β > 2 , q ∈ N . Let f 1 , . . . , f q ∈ W β 2 ( R d ) ∩ L ∞ ( R d ) and the function p ⋆ = ∑ q i =1 f 2 i . Let ε ∈ (0 , 1] and let η ∈ R d ++ . Let ϕ X,T be the feature map of the Gaussian kernel with bandwidth η and let H X ⊗ H T be the associated RKHS. Then there exists M ε ∈ S + ( H X ⊗ H T ) with rank( M ε ) ≤ q , such that for the representation ˆ p ( x, t ) = ϕ ⊤ X,T M ε ϕ X,T , following holds under assumption 2 on the coefficients of the FPE,

where | η | = det(diag( η )) , and ˜ C ′ and ˆ C depend only on β, d, ∥ f i ∥ W β 2 ( R d ) , ∥ f i ∥ L ∞ ( R d ) , ∥ ∥ ∥ ∂f i ∂x j ∥ ∥ ∥ L 2 ( R d ) , and ∥ ∥ ∥ ∂ 2 f i ∂x j x k ∥ ∥ ∥ 2 R d .

Even though, from theorem 1, we know that there exists a positive operator M ε : H X ⊗ H T → H X ⊗ H T such that ˆ p ( x, t ) = ϕ X,T ( x, t ) ⊤ M ε ϕ X,T ( x, t ) approximate the solution p ⋆ ( x, t ) of FPE from Theorem 1, it is not clear how to compute M ε as we do not have access of f j for j ∈ { 1 , 2 , · · · , q } beforehand. Hence, to get the final approximation error bound we need to further approximate ˆ p ( x, t ) with a PSD model of finite dimension ˜ p ( x, t ) which can be computed if we have the access to a finite number of base points ( x i , t i ) for i ∈ { 1 , · · · , m } in the domain ˜ X . Let us now construct a matrix A m as follows. Consider A m = K -1 ( X,T )( X,T ) ˜ ZM ˜ Z ⋆ K -1 ( X,T )( X,T ) , where K ( X,T )( X,T ) is a kernel matrix in R m × m such that [ K ( X,T )( X,T ) ] ij = ϕ X,T ( x i , t i ) ⊤ ϕ X,T ( x j , t j ) . Let us define the operator ˜ Z : H X ⊗ H T → R m such that ˜ Zu = ( ψ ( x 1 , t 1 ) ⊤ u, · · · , ψ ( x m , t m ) ⊤ u ) , and ˜ Z ⋆ α = ∑ m i =1 ψ ( x i , t i ) α i . We further define a projection operator ˜ P = ˜ Z ⋆ K -1 ( X,T )( X,T ) ˜ Z . Hence, we now define the approximation density ˜ p ( x, t ) as

In the main result of this section below, we show that ˜ p ( x, t ) is a good approximation for ˆ p ( x, t ) and hence for the true solution p ⋆ ( x, t ) .

Theorem 2. Let p ⋆ satisfy Assumptions 1 and 2. Let ε > 0 and β > 2 . There exists a Gaussian PSD model of dimension m ∈ N , i.e., ˜ p ( x, t ) = f (( x, t ) , A m , X, η ) , with A m ∈ S m + and X ∈ R m × d +1

and η m ∈ R d ++ , such that with m = O ( ε -( d +1) / ( β -2) (log 1 ε ) ( d +1) / 2 ) , we have,

Theorem 2 indicates that there is a positive semi-definite matrix A m ∈ R m × m such that ˜ p ( x, t ) = ψ ( x, t ) ⊤ A m ψ ( x, t ) is a good approximation of p ⋆ ( x, t ) . Next, we would see the usefulness of the above result in Theorem 2 in sampling from the solution of an FPE given the drift term µ satisfies the regularity condition as in [BRS16, Theorem 1.1] and µ ∈ W β 2 ( ˜ X ) . In this case, the drift function µ can be approximated by a function ˆ µ in a Gaussian RKHS, and the approximation guarantees are due to interpolation result in Lemma 4. The following result is the direct consequence of results in the Theorem 2 and Proposition 2.

Efficient sampling method Let ˆ µ ∈ H η be the approximation of µ such that ∥ µ -ˆ µ ∥ L 2 ( ˜ X ) = O ( ε ) , by Gaussian kernel approximation. Let us apply the construction of Thm.2 to ˆ µ .

Step 1: Find the best model ˆ A m by minimizing the l.h.s. of equation (12) over the set of m -dimensional Gaussian PSD models. Note that in that case, since everything is a linear combination of Gaussians (or products of Gaussians), we can compute the L 2 distance in closed form and minimize it over the positive semidefinite cone of m -dimensional matrices, obtaining ˆ A γ exactly.

Step 2: Denote the resulting PSD model by ˆ p γ ( x, t ) = f ( x, t ; ˆ A γ , η ) . At any time t of interest we sample from the probability ˆ p γ ( · , t ) via the sampling algorithm in [MFBR22].

Corollary 1 (Error of sampling) . Assume that µ ∈ W β 2 ( ˜ X ) satisfies the regularity condition as in [BRS16, Theorem 1.1]. For any time t , the algorithm described above provides i.i.d. samples from a probability ˜ p t that satisfies in expectation E t W 1 ( p ⋆ t , ˜ p t ) 2 = O ( ε 2 ) , where p ⋆ t := p ⋆ ( · , t ) is the probability at time t of a stochastic process with drift µ and diffusion D . Moreover, the cost of the algorithm is O ( m 2 d log( d/ε )) where m = O ( ε -d/β -2 (log(1 /ε )) d ) .

This result is proven in the appendix after theorem 2. It turns out that the parameter of the PSD model can be learned by a simple semi-definite program [RMFB20] which has a computational complexity of O ( m 3 . 5 log(1 /ε )) where m is the dimension of the model.

Remark. It is evident from the aforementioned result that our method excels in sampling from the Stochastic Differential Equation (SDE) while ensuring the particle density adheres to Assumption 1 within a bounded domain. In contrast, traditional sampling methods such as Langevin struggle to sample from this extensive class due to the absence of dissipativity, log-sobolev, and Poincaré-like conditions.

4.2 Approximating Solution of Fractional Fokker-Planck Equation

In this section, our aim is to approximate the solution p ⋆ ( x, t ) of a fractional Fokker-Planck equation, similar to the previous section. The equation replaces the Laplacian operator with a fractional Laplacian operator while keeping other terms unchanged. The fractional Laplacian operator, being non-local, is generally challenging to compute analytically. Therefore, approximating the fractional operator on a function has become a separate research direction. Below, we demonstrate that utilizing a PSD model-based representation is a natural choice for approximating the fractional operator acting on probability density. We present results for two cases: (i) a Gaussian kernel with the bandwidth parameter η , and (ii) a shift-invariant kernel represented using Bochner's theorem [Rud17].

Theorem 3. Consider probability density represented by a PSD model as p ( x ) = f ( x ; A,X,η ) = ∑ m i,j =1 A ij k ( x i , x ) k ( x j , x ) for some PSD matrix A , where X ∈ R m × d is the data matrix whose d

i -th row represents sample x i ∈ R for i ∈ { 1 , · · · , m } , then 1. if the kernel k is Gaussian with bandwidth parameter η , we have m where µ = 2 jη ( x i + x j ) , Σ = 4 ηI and C can be computed in the closed form.

if the kernel k have the bochner's theorem representation, i.e. k ( x -y ) = ∫ R d q ( ω ) e jω ⊤ ( x -y ) dω where q is a probability density, then m

where expectations are over ω ℓ , ω k ∼ q ( · ) .

We utilize the Fourier multiplier-based definition of the fractional Laplacian operator in Theorem 3. In both cases discussed in Theorem 3, empirical estimation of ( -∆) s p ( x ) , obtained from finite samples (empirical average), is sufficient for practical purposes. Shifting our focus from the representation of a non-local operator acting on a probability density, we now turn to the approximation result for the solution of a fractional Fokker-Planck equation in this section. We assume that the optimal solution p ⋆ ( x, t ) can be expressed as a sum of squares of q functions, as stated in assumption 1. To establish the approximation results for the fractional Fokker-Planck equation, we need to impose an additional mild assumption on f j for j ∈ 1 , · · · , q , in addition to assumption 1, which we state next.

Assumption 3. Let F [ f ]( · ) denotes the Fourier transform of a function f , then F [ f i ]( · ) ∈ L 1 ( ˜ X ) ∩ L ∞ ( ˜ X ) , and F [( -∆) s f i ]( · ) ∈ L 1 ( ˜ X ) for i ∈ { 1 , · · · q } .

Similar to the previous section, it requires two steps to obtain the approximation error bound. By similar construction as in section 4.1, we define an infinite dimensional positive operator M ε : H X ⊗ H T → H X ⊗ H T and based on this positive operator, we have a probability density ˆ p as ˆ p ( x, t ) = ϕ X,T ( x, t ) ⊤ M ε ϕ X,T ( x, t ) which is O ( ε ) approximation of p ⋆ ( x, t ) . The result is stated below in Theorem 4 and proven in Appendix C.3.

Theorem 4. Let β > 2 , q ∈ N . Let f 1 , . . . , f q satisfy assumptions 1 and 3 and the function p ⋆ = ∑ q i =1 f 2 i . Let ε ∈ (0 , 1] and let η ∈ R d ++ . Let ϕ X,T be the feature map of the Gaussian kernel with bandwidth η and let H X ⊗ H T be the associated RKHS. Then there exists M ε ∈ S + ( H X ⊗ H T ) with rank( M ε ) ≤ q , such that for the representation ˆ p ( x, t ) = ϕ ⊤ X,T M ε ϕ X,T , following holds under assumption 2 on the coefficients of the fractional FPE,

where | η | = det(diag( η )) , and ˜ C and ˆ C depend only on β, d and properties of f 1 , · · · , f q .

Major difficulty in proving the result in Theorem 4 arises from the challenge of effectively managing the fractional Laplacian term. To tackle this issue, we utilize the Fourier-based representation to gain control over the error associated with the fractional Laplacian. The details can be found in Lemma7, presented in Appendix C.1. As a next step, we would follow a similar procedure as in section 4.1 to obtain the final approximation bound. In the second part of the proof, we utilize Gagliardo-Nirenberg inequality [MP18] for fractional Laplacian to obtain the final approximation guarantee. In the next result, we show that for a particular choice of a PSD matrix A m , ˜ p ( x, t ) = ψ ( x, t ) ⊤ A m ψ ( x, t ) = ϕ X,T ( x, t ) ⊤ ˜ PM ˜ Pϕ X,T ( x, t ) is a good approximation for the true solution p ⋆ ( x, t ) of the fractional Fokker-Planck equation.

Theorem 5. Let p ⋆ satisfy Assumptions 1, 2, and 3. Let ε > 0 and β > 2 . There exists a Gaussian PSD model of dimension m ∈ N , i.e., ˜ p ( x, t ) = f (( x, t ) , A m , X, ϕ X,T ) , with A m ∈ S m + and X ∈ R m × d +1 and η m ∈ R d ++ , such that with m = O ( ε -( d +1) / ( β -2 s ) (log 1 ε ) ( d +1) / 2 ) , we have

Similar results as in Corollary 1 can be obtained for the case fractional Fokker-Planck equations as well. We can directly utilize results from [RZG¸ S23] to bound the Wasserstein metric between the solution of two fractional FPE whose drift terms are close in L ∞ metric. However, to find a PSD matrix A m from finite samples does require two separate sampling procedures which would contribute to the estimation error, (i) sampling to estimate the mean in the estimation of the fractional Laplacian and (ii) sampling of a finite number of data points from the data generating distribution.

Remark. As far as we know, Theorem 5 represents the first proof of approximation error bounds for the solution of the Fractional Fokker-Planck Equation when the approximated solution is a density function. This significant result allows for the utilization of the algorithm presented in [MFBR22] to sample from the approximate solution.

5 Conclusion

In this paper, we study the approximation properties of PSD models in modeling the solutions of FPEs and fractional FPEs. For the FPE, we show that a PSD model of size m = ˜ O ( ε -( d +1) / ( β -2) ) can approximate the true solution density up to order ε in L 2 metric where β is the order of smoothness of the true solution. Furthermore, we extend our result to the fractional FPEs to show that the required model size to achieve ε -approximation error to the solution density is m = ˜ O ( ε -( d +1) / ( β -2 s ) ) where s is the fractional order of the fractional Laplacian. In the process, we also show that PSD model-based representations for probability densities allow an easy way to approximate the fractional Laplacian operator (non-local operator) acting on probability density. As a future research direction, we would like to investigate and obtain a finite sample bound on the estimation error under regularity conditions on the drift term µ .

References

[AA20] Omar Abu Arqub. Numerical simulation of time-fractional partial differential equations arising in fluid flows via reproducing kernel method. International Journal of Numerical Methods for Heat &Fluid Flow , 30(11):4711-4733, 2020.

[AAS20] Omar Abu Arqub and Mohammed Al-Smadi. An adaptive numerical approach for the solutions of fractional advection-diffusion and dispersion equations in singular case under riesz's derivative operator. Physica A: Statistical Mechanics and its Applications , 540:123257, 2020.

[Ame14] William F Ames. Numerical methods for partial differential equations . Academic press, 2014.

[Arq18] Omar Abu Arqub. Numerical solutions for the robin time-fractional partial differential equations of heat and fluid flows based on the reproducing kernel algorithm. International Journal of Numerical Methods for Heat & Fluid Flow , 28(4):828-856, 2018.

[ASA19] Mohammed Al-Smadi and Omar Abu Arqub. Computational algorithm for solving fredholm time-fractional partial integrodifferential equations of dirichlet functions type with error estimates. Applied Mathematics and Computation , 342:280-294, 2019.

[Bar01] E Barkai. Fractional fokker-planck equation, solution, and application. Physical Review E , 63(4):046118, 2001.

[BCE + 22] Krishna Balasubramanian, Sinho Chewi, Murat A Erdogdu, Adil Salim, and Shunshi Zhang. Towards a theory of non-log-concave sampling: first-order stationarity guarantees for langevin monte carlo. In Conference on Learning Theory , pages 2896-2923. PMLR, 2022.

[BCRB22] Eloïse Berthier, Justin Carpentier, Alessandro Rudi, and Francis Bach. Infinite-dimensional sumsof-squares for optimal control. In 2022 IEEE 61st Conference on Decision and Control (CDC) , pages 577-582. IEEE, 2022.

[BDMP17] Nicolas Brosse, Alain Durmus, Éric Moulines, and Marcelo Pereyra. Sampling from a log-concave distribution with compact support with proximal langevin monte carlo. In Conference on learning theory , pages 319-342. PMLR, 2017.

[BEL15] Sebastien Bubeck, Ronen Eldan, and Joseph Lehec. Finite-time analysis of projected langevin monte carlo. Advances in Neural Information Processing Systems , 28, 2015.

[BH17] Jake Bouvrie and Boumediene Hamzi. Kernel methods for the approximation of some key quantities of nonlinear systems. Journal of Computational Dynamics , 4(1&2):1-19, 2017.

[BHJK20] Christian Beck, Martin Hutzenthaler, Arnulf Jentzen, and Benno Kuckuck. An overview on deep learning-based approximation methods for partial differential equations. arXiv preprint arXiv:2012.12348 , 2020.

[BKO + 96] Ted Belytschko, Yury Krongauz, Daniel Organ, Mark Fleming, and Petr Krysl. Meshless methods: an overview and recent developments. Computer methods in applied mechanics and engineering , 139(1-4):3-47, 1996.

[BQL + 22] Kaijun Bao, Xu Qian, Ziyuan Liu, Haifeng Wang, and Songhe Song. Function-valued rkhs-based operator learning for differential equations. arXiv preprint arXiv:2202.09488 , 2022.

[BRS16] Vladimir I Bogachev, Michael Röckner, and Stanislav V Shaposhnikov. Distances between transition probabilities of diffusions and applications to nonlinear fokker-planck-kolmogorov equations. Journal of Functional Analysis , 271(5):1262-1300, 2016.

[CDJB20] Niladri Chatterji, Jelena Diakonikolas, Michael I Jordan, and Peter Bartlett. Langevin monte carlo without smoothness. In International Conference on Artificial Intelligence and Statistics , pages 1716-1726. PMLR, 2020.

[CEL + 21] Sinho Chewi, Murat A Erdogdu, Mufan Bill Li, Ruoqi Shen, and Matthew Zhang. Analysis of langevin monte carlo from poincar \ 'e to log-sobolev. arXiv preprint arXiv:2112.12662 , 2021.

[CHOS21] Yifan Chen, Bamdad Hosseini, Houman Owhadi, and Andrew M Stuart. Solving and learning nonlinear pdes with gaussian processes. Journal of Computational Physics , 447:110668, 2021.

[CLL21] Ziang Chen, Jianfeng Lu, and Yulong Lu. On the representation of solutions to elliptic pdes in barron spaces. Advances in neural information processing systems , 34:6454-6465, 2021.

[CMW + 21] Shengze Cai, Zhiping Mao, Zhicheng Wang, Minglang Yin, and George Em Karniadakis. Physicsinformed neural networks (pinns) for fluid mechanics: A review. Acta Mechanica Sinica , 37(12):1727-1738, 2021.

[Dal17] Arnak S Dalalyan. Theoretical guarantees for approximate sampling from smooth and log-concave densities. Journal of the Royal Statistical Society. Series B (Statistical Methodology) , pages 651-676, 2017.

[DK19] Arnak S Dalalyan and Avetik Karagulyan. User-friendly guarantees for the langevin monte carlo with inaccurate gradient. Stochastic Processes and their Applications , 129(12):5278-5311, 2019.

[DM14] Mehdi Dehghan and Vahid Mohammadi. The numerical solution of fokker-planck equation with radial basis functions (rbfs) based on the meshless technique of kansas approach and galerkin method. Engineering Analysis with Boundary Elements , 47:38-63, 2014.

[DM17] Alain Durmus and Eric Moulines. Nonasymptotic convergence analysis for the unadjusted langevin algorithm. 2017.

[DM19] Alain Durmus and Eric Moulines. High-dimensional bayesian inference via the unadjusted langevin algorithm. 2019.

[DMM19] Alain Durmus, Szymon Majewski, and Bła˙ zej Miasojedow. Analysis of langevin monte carlo via convex optimization. The Journal of Machine Learning Research , 20(1):2666-2711, 2019.

[DRM22] Tim De Ryck and Siddhartha Mishra. Generic bounds on the approximation error for physicsinformed (and) operator learning. arXiv preprint arXiv:2205.11393 , 2022.

[Dua15] Jinqiao Duan. An introduction to stochastic dynamics , volume 51. Cambridge University Press, 2015.

[EH21] Murat A Erdogdu and Rasa Hosseinzadeh. On the convergence of langevin monte carlo: The interplay between tail growth and smoothness. In Conference on Learning Theory , pages 17761822. PMLR, 2021.

[Eva22] Lawrence C Evans. Partial differential equations , volume 19. American Mathematical Society, 2022.

[Fas07] Gregory E Fasshauer. Meshfree approximation methods with MATLAB , volume 6. World Scientific, 2007.

[FF15] Bengt Fornberg and Natasha Flyer. Solving pdes with radial basis functions. Acta Numerica , 24:215-258, 2015.

[FGN13] Xiaobing Feng, Roland Glowinski, and Michael Neilan. Recent developments in numerical methods for fully nonlinear second order partial differential equations. siam REVIEW , 55(2):205267, 2013.

[GW79] Ian Gladwell and R Wait. A survey of numerical methodes for partial differential equations. Oxford: Clarendon Press , 1979.

[HF08] Isom Herron and Michael Foster. Partial differential equations in fluid dynamics . Cambridge Univ. Press, 2008.

[HJE18] Jiequn Han, Arnulf Jentzen, and Weinan E. Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences , 115(34):8505-8510, 2018.

[HKRC18] Ya-Ping Hsieh, Ali Kavis, Paul Rolland, and Volkan Cevher. Mirrored langevin dynamics. Advances in Neural Information Processing Systems , 31, 2018.

[HMW21] Lu-Jing Huang, Mateusz B Majka, and Jian Wang. Approximation of heavy-tailed distributions via stable-driven sdes. 2021.

[Ise09] Arieh Iserles. A first course in the numerical analysis of differential equations . Number 44. Cambridge university press, 2009.

[KKL + 21] George Em Karniadakis, Ioannis G Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physics-informed machine learning. Nature Reviews Physics , 3(6):422-440, 2021.

[KLY21] Yuehaw Khoo, Jianfeng Lu, and Lexing Ying. Solving parametric pde problems with artificial neural networks. European Journal of Applied Mathematics , 32(3):421-435, 2021.

[KPRS22] Gitta Kutyniok, Philipp Petersen, Mones Raslan, and Reinhold Schneider. A theoretical analysis of deep neural networks and parametric pdes. Constructive Approximation , 55(1):73-125, 2022.

[KRP12] S Kazem, JA Rad, and K Parand. Radial basis functions methods for solving fokker-planck equation. Engineering Analysis with Boundary Elements , 36(2):181-189, 2012.

[Lam21] Andrew Lamperski. Projected stochastic gradient langevin algorithms for constrained sampling and non-convex learning. In Conference on Learning Theory , pages 2891-2937. PMLR, 2021.

[LAT04] Fawang Liu, Vo Anh, and Ian Turner. Numerical solution of the space fractional fokker-planck equation. Journal of Computational and Applied Mathematics , 166(1):209-219, 2004.

[LCL + 21] Yiping Lu, Haoxuan Chen, Jianfeng Lu, Lexing Ying, and Jose Blanchet. Machine learning for elliptic pdes: fast rate generalization bound, neural scaling law and minimax optimality. arXiv preprint arXiv:2110.06897 , 2021.

[LL22] Jianfeng Lu and Yulong Lu. A priori generalization error analysis of two-layer neural networks for solving high dimensional schrödinger eigenvalue problems. Communications of the American Mathematical Society , 2(01):1-21, 2022.

[LLMD18] Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong. Pde-net: Learning pdes from data. In International conference on machine learning , pages 3208-3216. PMLR, 2018.

[LLW21] Jianfeng Lu, Yulong Lu, and Min Wang. A priori generalization analysis of the deep ritz method for solving high dimensional elliptic equations. arXiv preprint arXiv:2101.01708 , 2021.

[LT00] Irena Lasiecka and Roberto Triggiani. Control theory for partial differential equations: Volume 1, Abstract parabolic systems: Continuous and approximation theories , volume 1. Cambridge University Press, 2000.

[MCF15] Yi-An Ma, Tianqi Chen, and Emily Fox. A complete recipe for stochastic gradient mcmc. Advances in neural information processing systems , 28, 2015.

[MFBR20] Ulysse Marteau-Ferey, Francis Bach, and Alessandro Rudi. Non-parametric models for nonnegative functions. Advances in neural information processing systems , 33:12816-12826, 2020.

[MFBR22] Ulysse Marteau-Ferey, Francis Bach, and Alessandro Rudi. Sampling from arbitrary functions via psd models. In International Conference on Artificial Intelligence and Statistics , pages 2823-2861. PMLR, 2022.

[MP18] Carlo Morosi and Livio Pizzocchero. On the constants for some fractional gagliardo-nirenberg and sobolev inequalities. Expositiones Mathematicae , 36(1):32-77, 2018.

[MVB + 21] Boris Muzellec, Adrien Vacher, Francis Bach, François-Xavier Vialard, and Alessandro Rudi. Near-optimal estimation of smooth transport maps with kernel sums-of-squares. arXiv preprint arXiv:2112.01907 , 2021.

[NF21] Christopher Nemeth and Paul Fearnhead. Stochastic gradient markov chain monte carlo. Journal of the American Statistical Association , 116(533):433-450, 2021.

[Nir11] Louis Nirenberg. On elliptic partial differential equations. In Il principio di minimo e sue applicazioni alle equazioni funzionali , pages 1-48. Springer, 2011.

[NSGR19] Thanh Huy Nguyen, Umut Simsekli, Mert Gurbuzbalaban, and Gaël Richard. First exit time analysis of stochastic gradient descent under heavy-tailed gradient noise. Advances in neural information processing systems , 32, 2019.

[NSR19] Than Huy Nguyen, Umut Simsekli, and Gaël Richard. Non-asymptotic analysis of fractional langevin monte carlo for non-convex optimization. In International Conference on Machine Learning , pages 4810-4819. PMLR, 2019.

[Pav14] Grigorios A Pavliotis. Stochastic processes and applications: diffusion processes, the FokkerPlanck and Langevin equations , volume 60. Springer, 2014.

[PMB13] Lukas Pichler, Arif Masud, and Lawrence A Bergman. Numerical solution of the fokker-planck equation by finite difference and finite element methods-a comparative study. Computational Methods in Stochastic Dynamics: Volume 2 , pages 69-85, 2013.

[PR20] Vivek G Patel and Nikunj V Rachchh. Meshless method-review on recent developments. Materials today: proceedings , 26:1598-1603, 2020.

[PRDVR20] Nicolò Pagliana, Alessandro Rudi, Ernesto De Vito, and Lorenzo Rosasco. Interpolation and learning with scale dependent kernels. arXiv preprint arXiv:2006.09984 , 2020.

[RC21] Alessandro Rudi and Carlo Ciliberto. Psd representations for effective probability models. Advances in Neural Information Processing Systems , 34, 2021.

[RMFB20] Alessandro Rudi, Ulysse Marteau-Ferey, and Francis Bach. Finding global minima via kernel approximations. arXiv preprint arXiv:2012.11978 , 2020.

[RPK19] Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics , 378:686-707, 2019.

[RR96] Hannes Risken and Hannes Risken. Fokker-planck equation . Springer, 1996.

[RRT17] Maxim Raginsky, Alexander Rakhlin, and Matus Telgarsky. Non-convex learning via stochastic gradient langevin dynamics: a nonasymptotic analysis. In Conference on Learning Theory , pages 1674-1703. PMLR, 2017.

[Rud17] Walter Rudin. Fourier analysis on groups . Courier Dover Publications, 2017.

[RZ10] Christian Rieger and Barbara Zwicknagl. Sampling inequalities for infinitely smooth functions, with applications to interpolation and machine learning. Advances in Computational Mathematics , 32(1):103-129, 2010.

[RZG¸ S23] Anant Raj, Lingjiong Zhu, Mert Gürbüzbalaban, and Umut ¸ Sim¸ sekli. Algorithmic stability of heavy-tailed sgd with general loss functions. arXiv preprint arXiv:2301.11885 , 2023.

[SAA20] Oleg Szehr, Dario Azzimonti, and Laura Azzimonti. An exact kernel framework for spatio-temporal dynamics. arXiv preprint arXiv:2011.06848 , 2020.

[Sch07] R Schaback. Kernel-based meshless methods. Lecture Notes for Taught Course in Approximation Theory. Georg-August-Universität Göttingen , 2007.

[ ¸ Sim17] Umut ¸ Sim¸ sekli. Fractional langevin monte carlo: Exploring lévy driven stochastic differential equations for markov chain monte carlo. In International Conference on Machine Learning , pages 3200-3209. PMLR, 2017.

[SR20] Adil Salim and Peter Richtárik. Primal dual interpretation of the proximal stochastic gradient langevin algorithm. Advances in Neural Information Processing Systems , 33:3786-3796, 2020.

[SS + 16] Saburou Saitoh, Yoshihiro Sawano, et al. Theory of reproducing kernels and applications . Springer, 2016.

[SS22] BSepehrian and Z Shamohammadi. Solution of the liouville-caputo time-and riesz space-fractional fokker-planck equation via radial basis functions. Asian-European Journal of Mathematics , 15(11):2250195, 2022.

[SSDE20] Umut Simsekli, Ozan Sener, George Deligiannidis, and Murat A Erdogdu. Hausdorff dimension, heavy tails, and generalization in neural networks. Advances in Neural Information Processing Systems , 33:5138-5151, 2020.

[SSG19] Umut Simsekli, Levent Sagun, and Mert Gurbuzbalaban. A tail-index analysis of stochastic gradient noise in deep neural networks. In International Conference on Machine Learning , pages 5827-5837. PMLR, 2019.

[Ste21] George Stepaniants. Learning partial differential equations in reproducing kernel hilbert spaces. arXiv preprint arXiv:2108.11580 , 2021.

[STL96] Gennady Samorodnitsky, Murad S Taqqu, and RW Linde. Stable non-gaussian random processes: stochastic models with infinite variance. Bulletin of the London Mathematical Society , 28(134):554555, 1996.

[SW06] Robert Schaback and Holger Wendland. Kernel techniques: from machine learning to meshless methods. Acta numerica , 15:543-639, 2006.

[Tad12] Eitan Tadmor. A review of numerical methods for nonlinear partial differential equations. Bulletin of the American Mathematical Society , 49(4):507-554, 2012.

[Trö10] Fredi Tröltzsch. Optimal control of partial differential equations: theory, methods, and applications , volume 112. American Mathematical Soc., 2010.

[TSIG22] Armin Tabandeh, Neetesh Sharma, Leandro Iannacone, and Paolo Gardoni. Numerical solution of the fokker-planck equation using physics-based mixture models. Computer Methods in Applied Mechanics and Engineering , 399:115424, 2022.

[UHK18] Sabir Umarov, Marjorie Hahn, and Kei Kobayashi. Beyond the Triangle: Brownian Motion, Itô Calculus, and Fokker-Planck Equation-Fractional Generalizations . World Scientific, 2018.

[VMR + 21] Adrien Vacher, Boris Muzellec, Alessandro Rudi, Francis Bach, and Francois-Xavier Vialard. A dimension-free computational upper-bound for smooth optimal transport estimation. In Conference on Learning Theory , pages 4143-4173. PMLR, 2021.

[Wen04] Holger Wendland. Scattered data approximation , volume 17. Cambridge university press, 2004.

[WWLZ22] Jia-Li Wei, Guo-Cheng Wu, Bao-Qing Liu, and Zhengang Zhao. New semi-analytical solutions of the time-fractional fokker-planck equation by the neural network method. Optik , 259:168896, 2022.

[XCZG18] Pan Xu, Jinghui Chen, Difan Zou, and Quanquan Gu. Global convergence of langevin dynamics based algorithms for nonconvex optimization. Advances in Neural Information Processing Systems , 31, 2018.

[XZL + 20] Yong Xu, Hao Zhang, Yongge Li, Kuang Zhou, Qi Liu, and Jürgen Kurths. Solving fokkerplanck equation using deep learning. Chaos: An Interdisciplinary Journal of Nonlinear Science , 30(1):013133, 2020.

[Y + 18] Bing Yu et al. The deep ritz method: a deep learning-based numerical algorithm for solving variational problems. Communications in Mathematics and Statistics , 6(1):1-12, 2018.

[Yan13] Limei Yan. Numerical solutions of fractional fokker-planck equations using iterative laplace transform method. In Abstract and applied analysis , volume 2013. Hindawi, 2013.

[ZDL22] Jiayu Zhai, Matthew Dobson, and Yao Li. A deep learning method for solving fokker-planck equations. In Mathematical and Scientific Machine Learning , pages 568-597. PMLR, 2022.

[ZFM + 20] Pan Zhou, Jiashi Feng, Chao Ma, Caiming Xiong, Steven Chu Hong Hoi, et al. Towards theoretically understanding why sgd generalizes better than adam in deep learning. Advances in Neural Information Processing Systems , 33:21285-21296, 2020.

[Zho08] Ding-Xuan Zhou. Derivative reproducing properties for kernel methods in learning theory. Journal of computational and Applied Mathematics , 220(1-2):456-463, 2008.

[ZL22] Yuping Zheng and Andrew Lamperski. Constrained langevin algorithms with l-mixing external random variables. arXiv preprint arXiv:2205.14192 , 2022.

[ZZ22] Xiaolong Zhang and Xicheng Zhang. Ergodicity of supercritical sdes driven by alpha-stable processes and heavy-tailed sampling. arXiv preprint arXiv:2201.10158 , 2022.

Appendix

A Existence of an appropriate PSD matrix M ε (Fokker-Planck Equation)

A.1 Useful Results

Consider the mollifier function g defined as in [RC21].

where J d/ 2 is the Bessel function of the first kind of order d/ 2 and V d = ∫ ∥ x ∥≤ 1 dx = π d/ 2 Γ( d/ 2+1) . g v ( x ) = v -d g ( x/v ) .

Lemma 1 ([RC21], Lemma D.3) . The function g defined above satisfies g ∈ L 1 ( R d ) ∩ L 2 ( R d ) and ∫ g ( x ) dx = 1 . Moreover, for any ω ∈ R d , we have

The following proposition provides a useful characterization of the space W β 2 ( R d )

Proposition 4 (Characterization of the Sobolev space W k 2 ( R d ) , [Wen04]) . Let k ∈ N . The norm of the Sobolev space ∥ · ∥ W k 2 ( R d ) is equivalent to the following norm

and satisfies

Moreover, when k > d/ 2 , then W k 2 ( R d ) is a reproducing kernel Hilbert space.

Lemma 2. Consider the definition of g v ( x ) = v -d g ( x/v ) where g is defined in equation (13) . Let β > 2 , q ∈ N . Let f 1 , . . . , f q ∈ W β 2 ( R d ) ∩ L ∞ ( R d ) and the function p ⋆ = ∑ q i =1 f 2 i . Let us assume that the Assumption 2 is also satisfied. Let ε ∈ (0 , 1] and let η ∈ R d ++ . Let ϕ X,T be the feature map of the Gaussian kernel with bandwidth η and let H X ⊗ H T be the associated RKHS. Then there exists M ε ∈ S + ( H X ⊗ H T ) with rank( M ε ) ≤ q , such that for the representation ˆ p ( x, t ) = ϕ ⊤ X,T M ε ϕ X,T , following holds for v > 0 and k ∈ { 1 , · · · d } ,

Hence,

This gives,

Proof. We have g defined as in [RC21].

where J d/ 2 is the Bessel function of the first kind of order d/ 2 and V d = ∫ ∥ x ∥≤ 1 dx = π d/ 2 Γ( d/ 2+1) . g v ( x ) = v -d g ( x/v ) . We use result of lemma 1 to prove the statement. For a function f ∈ W β 2 ( R d ) ,

Hence,

The same result will also hold for ∂ ∂t without loss of generality. Hence,

We now bound ∥ ∥ ∥ ∂p ⋆ ( x,t ) ∂t -∂ ˆ p ( x,t ) ∂t ∥ ∥ ∥ L r ( R d ) and ∥ ∥ ∥ ∂p ⋆ ( x,t ) ∂x i -∂ ˆ p ( x,t ) ∂x i ∥ ∥ ∥ L r ( R d ) .

As in [RC21], let us denote, f i ⋆ g v as f i,v . We consider

Hence, we have

Similarly,

Hence,

If we choose

Similarly,

∥ ∥ ∥ ∥ ∂p ⋆ ( x, t ) ∂t -∂ ˆ p ( x, t ) ∂t ∥ ∥ ∥ ∥ L 2 ( R d ) = 2 q ∑ i =1 ∥ ∥ ∥ ∥ ∂f i ∂t -∂f i,v ∂t ∥ ∥ ∥ ∥ L 2 ( R d ) ∥ f i ∥ L ∞ ( R d ) +2 q ∑ i =1 ∥ f i -f i,v ∥ L 2 ( R d ) ∥ ∥ ∥ ∥ ∂f i,v ∂t ∥ ∥ ∥ ∥ L ∞ ( R d ) . Now,

Hence,

If we choose

where C 1 = ∑ q i =1 ∥ f i ∥ W β 2 ( R d ) ∥ f i ∥ L 2 ( R d ) and C 2 = ∑ q i =1 ∥ f i ∥ W β 2 ( R d ) ∥ ∥ ∥ ∂f i ∂t ∥ ∥ ∥ L 2 ( R d ) ∥ g ∥ L 1 ( R d ) . Hence,

where C 1 = ∑ q i =1 ∥ f i ∥ W β 2 ( R d ) ∥ f i ∥ L ∞ ( R d ) and C 2 = ∑ q i =1 ∥ f i ∥ W β 2 ( R d ) ∥ ∥ ∥ ∂f i ∂t ∥ ∥ ∥ L ∞ ( R d ) ∥ g ∥ L 1 ( R d ) . Hence,

Similarly,

and

Lemma 3. Consider the definition of g v ( x ) = v -d g ( x/v ) where g is defined in equation (13) . Let β > 2 , q ∈ N . Let f 1 , . . . , f q ∈ W β 2 ( R d ) ∩ L ∞ ( R d ) and the function p ⋆ = ∑ q i =1 f 2 i . Let us assume that the Assumption 2 is also satisfied. Let ε ∈ (0 , 1] and let η ∈ R d ++ . Let ϕ X,T be the feature map of the Gaussian kernel with bandwidth η and let H X ⊗ H T be the associated RKHS. Then there exists M ε ∈ S + ( H X ⊗ H T ) with rank( M ε ) ≤ q , such that for the representation ˆ p ( x, t ) = f ( x, t ) = ϕ ⊤ X,T M ε ϕ X,T , following holds for all i, j ∈ { 1 , · · · d } .,

and

Proof. We have g defined as in [RC21].

where J d/ 2 is the Bessel function of the first kind of order d/ 2 and V d = ∫ ∥ x ∥≤ 1 dx = π d/ 2 Γ( d/ 2+1) . g v ( x ) = v -d g ( x/v ) . We use result of lemma 1 to prove the statement. We have, for a function f ∈ W β 2 ( R d )

Hence,

Now we have,

Similarly,

Now,

As in [RC21], let us denote, f i ⋆ g v as f i,v . We consider

Let's consider all the terms separately.

Hence,

Hence, if we choose,

where C 1 = (1 + ∥ g ∥ L 1 ( R d ) ) ∑ q m =1 ∥ f ∥ W β 2 ( R d ) ∥ ∥ ∥ ∂f m ∂x j ∥ ∥ ∥ L 2 ( R d ) , C 2 = ∑ q m =1 ∥ f ∥ W β 2 ( R d ) ∥ f m ∥ L 2 ( R d ) and C 3 = ∑ q m =1 ∥ f m ∥ W β 2 ( R d ) ∥ ∥ ∥ ∂ 2 f m ∂x i ∂x j ∥ ∥ ∥ L 2 ( R d ) ∥ g ∥ L 1 ( R d ) , then,

This gives,

In similar way, we can also obtain bound for L 2 metric. Now we have,

Let us now compute the following,

and

Similarly,

Combining everything together, we have

Hence, we get the final result by replacing f ( x, t ) with ˆ p ( x, t ) .

A.2 Approximation Result by a PSD Model (Proof of Theorem 1)

Proof. Now, we just need to collect results from Lemmas 2 and 3 to get the final bound. Let us assume that there exist an optimal density function p ⋆ ( x, t ) which is the solution of fokkerplanck equation. Hence, We would like to show that there exists a psd operator M ε such that if ˆ p ( x, t ) = f ( x, t ) = ϕ X,T ( x, t ) ⊤ M ε ϕ X,T ( x, t ) such that we can obtain bound for following three quantities for r ∈ { 1 , 2 } :

If the above three condition holds, then, we can show that for r ∈ { 1 , 2 } ,

We have from Lemma 2 and equation (26), if we choose

From [RC21, Theorem D.4 (Eq. D.25) ], we have,

This proves the first part. Going further,

Now,

From [RC21, Theorem D.4 (Eq. D.25) ], we have,

And previously, we showed in lemma 2

If we choose,

This gives,

We can bound the trace of the matrix as follows. We have,

v

= min

( (

2

β

-

1

ε

16

πC

1

2

β

ε

+1

C

2

2

β

ε

+1

C

3

where C 1 = ∑ q i =1 ∥ f i ∥ W β 2 ( R d +1 ) ∥ f i ∥ L ∞ ( R d +1 ) , C 2 = ∑ q i =1 ∥ f i ∥ W β 2 ( R d +1 ) ∥ ∥ ∥ ∂f i ∂t ∥ ∥ ∥ L ∞ ( R d +1 ) ∥ g ∥ L 1 ( R d +1 ) and C 3 = (1 + ∥ g ∥ L 1 ( R d +1 ) ) ∑ q i =1 ∥ f i ∥ W β 2 ( R d +1 ) ∥ f i ∥ L 2 ( R d +1 ) , then, we have

We showed in lemma 3 that

Hence, if we choose,

where C 1 = (1 + ∥ g ∥ L 1 ( R d +1 ) ) ∑ q m =1 ∥ f ∥ W β 2 ( R d +1 ) ∥ ∥ ∥ ∂f m ∂x j ∥ ∥ ∥ L ∞ ( R d +1 ) , C 2 = ∑ q m =1 ∥ f ∥ W β 2 ( R d +1 ) ∥ f m ∥ L ∞ ( R d +1 ) and C 3 = ∑ q m =1 ∥ f m ∥ W β 2 ( R d +1 ) ∥ ∥ ∥ ∂ 2 f m ∂x i ∂x j ∥ ∥ ∥ L ∞ ( R d +1 ) ∥ g ∥ L 1 ( R d +1 ) , then,

From [RC21, Equation D.21], we have

Since, we choose,

)

1

-

(

)

1

β

(

)

1

β

)

β

1

,

,

where ˜ C = 96 π 2 2 β -2 C , ˜ C ′ = 89 ˜ C 2 β -2 , and ˆ C = π -d/ 2 2 2 β ∑ q i =1 ∥ f i ∥ 2 W β 2 ( R d +1 ) max ( 1 , 3 -2 β ˜ C 2 β β -2 ) .

A.3 Compression Result

Let X ⊂ R d be an open set with Lipschitz boundary, contained in the hypercube [ -R,R ] d with R > 0 . Given ˜ x 1 , . . . , ˜ x m ∈ X be m points in [ -R,R ] d . Define the base point matrix ˜ X ∈ R m × d to be the matrix whose j -th row is the point ˜ x j . The following result holds. We introduce the so called fill distance [Wen04]

Now, we state some useful results below.

A.4 Useful Results

Lemma 4 ([RC21], Lemma C.1 ) . Let T = ( -R,R ) d and η ∈ R d ++ . Let u ∈ H η satisfying u (˜ x 1 ) = · · · = u (˜ x m ) = 0 . There exists three constants c, C, C ′ depending only on d (and in particular, independent from R,η, u, ˜ x 1 , . . . , ˜ x m ), such that, when h ≤ σ/C ′ , then,

with q η = det( 1 η + diag( η )) -1 / 4 and σ = min( R, 1 √ η + ) and η + = max i =1 ,...,d η i .

Lemma 5 (Norm of function's derivative with scattered zeros) . Let T = ( -R,R ) d and η ∈ R d ++ . Let u ∈ H η satisfying u (˜ x 1 ) = · · · = u (˜ x m ) = 0 . There exists three constants c, C, C ′ depending only on d (and in particular, independent from R,η, u, ˜ x 1 , . . . , ˜ x m ), such that, when h ≤ σ/C ′ , then,

with G be some positive constant and σ = min( R, 1 √ η + ) and η + = max i =1 ,...,d η i .

Proof. Here, we utilize Gagliardo-Nirenberg inequality [Nir11]. We have, for 0 < k < ∞ , 0 ≤ j < ∞ and ϑ ( j, m ) = j k + d 2 k < 1 , then

where G = 1 2 d/ 2 π d/ 4 -1 / 2 √ Γ( d/ 2)(1 -ϑ ) 1 -ϑ ϑ ϑ k sin πϑ .

Considering the case for j = 1 and j = 2 , we have,

From the definition,

Hence,

Now, we make similar choice of the parameter h and k as in [RC21]. If we choose, C 3 = 1 max(2 B d ,B ′ d ) , h ≤ C 3 d +4 min( R, 1 / √ η + ) , k = ⌊ s ⌋ and s = C 3 h min( R, 1 / √ η + ) , then we have hk ≤ R B ′ d and 2 √ η + B d hk ≤ 1 . Moreover, 1 √ k ! ≤ e -k 2 log k 2 . If we also assume that G = max( G 1 , G 2 ) , then for j = { 1 , 2 }

The final result is obtained by writing s/ 4 = cσ/h with σ = min( R, 1 / √ η + ) and c = C 3 / 4 .

Lemma 6 (Lemma 3, page 28 [PRDVR20]) . Let X ⊂ R d with non-zero volume. Let H be a reproducing kernel Hilbert space on X , associated to a continuous uniformly bounded feature map ϕ : X → H . Let A : H → H be a bounded linear operator. Then,

In particular, if X ⊂ R d is a non-empty open set, then sup x ∈ X ∥ Aϕ ( x ) ∥ H ≤ sup ∥ f ∥ H ≤ 1 ∥ A ∗ f ∥ L ∞ ( X ) .

where G 1 = 1 2 d/ 2 π d/ 4 -1 / 2 √ Γ( d/ 2)(1 -d +2 2 k ) 1 -d +2 2 k ( d +2 2 k ) d +2 2 k k sin π ( d +2 2 k ) and

G 2 = 1 2 d/ 2 π d/ 4 -1 / 2 √ Γ( d/ 2)(1 -d +4 2 k ) 1 -d +4 2 k ( d +4 2 m ) d +4 2 k k sin π ( d +4 2 k ) . We here have assumed that k > d 2 +2 .

From the defeinition, we have

Now by Theorem 4.3 of [RZ10], we have for some constants B d and B ′ d depending only on d,

when kh ≤ R B ′ d . Hence, we have,

Theorem 6 ( Theorem C.3 [RC21]) . Let R > 0 , η ∈ R d ++ , m ∈ N . Let X ⊆ T = ( -R,R ) d be a non-empty open set and let ˜ x 1 , . . . , ˜ x m be a set of distinct points. Let h > 0 be the fill distance associated to the points w.r.t T (defined in eq. (34) ). Let ˜ P : H η → H η be the associated projection operator (see definition in eq. (38) ). There exists three constants c, C, C ′ , such that, when h ≤ σ/C ′ ,

Here q η = det( 1 η + diag( η )) -1 / 4 and σ = min( R, 1 √ η + ) , η + = max i η i . The constants c, C ′ , C ′′ depend only on d and, in particular, are independent from R,η, ˜ x 1 , . . . , ˜ x m .

Similar to the results in theorem 6, we will obtain results for derivatives.

Theorem 7. Let R > 0 , η ∈ R d ++ , m ∈ N . Let X ⊆ T = ( -R,R ) d be a non-empty open set and let ˜ x 1 , . . . , ˜ x m be a set of distinct points. Let h > 0 be the fill distance associated to the points w.r.t T (defined in eq. (34) ). Let ˜ P : H η → H η be the associated projection operator (see definition in eq. (38) ). There exists three constants c, C, C ′ , such that, when h ≤ σ/C ′ ,

for some positive constant C 1 . Here q η = det( 1 η + diag( η )) -1 / 4 and σ = min( R, 1 √ η + ) , η + = max i η i . The constants c, C ′ , C ′′ depend only on d and, in particular, are independent from R,η, ˜ x 1 , . . . , ˜ x m .

Proof. If the function f lies in the RKHS, then using the property of the projection operator ˜ P , we have

Hence, D j f -˜ PD j f (˜ x i ) = 0 for all i ∈ { 1 , · · · , m } . By lemma 4, we know that there exist three constants c, C, C ′ depending only on d such that when h ≤ σ/C ′ we have that the following holds ∥ u ∥ L ∞ ( T ) ≤ Cq η e -c σ h log c σ h , for any u ∈ H η such that u (˜ x 1 ) = · · · = u (˜ x m ) = 0 . Since, for any u ∈ H η , we have that D j f -˜ PD j f belongs to H η and satisfies such property, we can apply lemma 4 with u = ( I -˜ P ) D j f , obtaining, under the same assumption on h ,

where we used the fact that ∥ ( I -˜ P ) D j f ∥ H η ≤ ∥ I -˜ P ∥∥ D j f ∥ H η and ∥ I -˜ P ∥ ≤ 1 , since P is a projection operator and so also I -P satisfies this property. Derivative reproducing properties for kernel can be looked at in [Zho08]. The final result is obtained by applying lemma 6 with A = I -˜ P , from which we have

for some positive C 1 . The same result can also be obtained by the use of Lemma 5 with a condition that f is d/ 2 + 1 differentiable function which is a slightly weaker condition on the order of smoothness.

A.5 Compression of a PSD model for Fokker-Planck

We have (˜ x 1 , ˜ t 1 ) , ˜ x 2 , · · · (˜ x m , ˜ t m ) ∈ X × T which is an open bounded subset with Lipschitz boundary of the cube [ -R,R ] d × [0 , T ] . ˜ X be the base point matrix whose j-rows are the points ˜ x j . Let us now consider the model p = f ( · , M, ψ ) and ˜ p = f ( · ; A m , ˜ X,ψ ) where A m = K -1 ( X,T )( X,T ) ˜ ZM ˜ Z ⋆ K -1 ( X,T )( X,T ) .

and

Hence, for any A ∈ R m × m

Some properties associated with it,

Theorem 8. Let η ∈ R d ++ and let M ∈ S + ( H η ) . Let ˜ X be an open bounded subset with the Lipschitz boundary of the cube [ -R,R ] d × [0 , R ] , R > 0 . Let ˜ x 1 , . . . , ˜ x m ∈ ˜ X and ˜ X be the base point matrix whose j -rows are the points ˜ x j with j = 1 , . . . , m . Consider the model p = f ( · ; M , ψ ) and the the compressed model ˜ p = f ( · ; A m , ˜ X,η ) with

where ˜ Z : H η → R m is defined in eq. (36) in terms of ˜ X m . Let h be the fill distance (defined in eq. (34) ) associated to the points ˜ x 1 , . . . , ˜ x m . The there exist three constants c, C, C ′ depending only on d such that, when h ≤ σ/C ′ , with σ = min( R, 1 / √ η + ) , η + = max i =1 ,...,d η i , q η = det( 1 η + diag( η )) -1 / 4 , then we have

Proof. We have,

Let us also define associated projection operator,

Let now consider every term separately.

Similarly,

Finally,

We have,

Since | a ⊤ ABAa | ≤ ∥ Aa ∥ 2 H ∥ B ∥ and | a ⊤ ABa | ≤ ∥ Aa ∥ H ∥ B 1 / 2 ∥∥ B 1 / 2 a ∥ H , for any a in a Hilbert space H and for A,B bounded symmetric linear operators with B ∈ S + ( H ) , by bounding the terms of the equation above, we have for any x ∈ R d ,

where c M = ∥ M ∥ and we denoted by ˜ u 1 ( x ) the quantity ˜ u 1 ( x ) = ∥ ( I -˜ P ) ψ ( x, t ) ∥ H η and we noted that ∥ M 1 / 2 ψ ( x, t ) ∥ 2 H η = ψ ( x, t ) ⊤ Mψ ( x, t ) = f ( x ; M,ψ ( x, t )) . Now,

Following similar steps, we get

where ˜ f x j x i ( x ; M,ψ ) = ∂ 2 ψ ( x,t ) ∂x j x i ⊤ M ∂ 2 ψ ( x,t ) ∂x j x i and ˜ u x j x i ( x ) = ∥ ∥ ∥ ( I -˜ P ) ∂ 2 ψ ( x,t ) ∂x j x i ∥ ∥ ∥ H η . And,

Hence, combining everything together, we have:

Now, combining everything together, we get

Now from theorems 6 and 7, we have that when the fill distance h < σ C ′ with σ = min( R, 1 τ ) , then ∥ ˜ u 1 ∥ L ∞ ( X ) ≤ Cq η e -cσ h log cσ h , ∥ ˜ u t ∥ L ∞ ( X ) ≤ Cq η e -cσ h log cσ h , ∥ ˜ u x i ∥ L ∞ ( X ) ≤ Cq η e -cσ h log cσ h (48) and ∥ ˜ u x i x j ∥ L ∞ ( X ) ≤ Cq η e -cσ h log cσ h for all i ∈ { 1 , · · · , d } , j ∈ { 1 , · · · , d } and some constant C, c, C ′ depending only on d . Hence, we have

A.6 Final Approximation Bound (Proof of Theorem 2)

Proof. From the previous result in theorem 1, we know that there exists an M ε ∈ S + ( H X ⊗ H T ) with rank( M ε ) ≤ q , such that for the representation ˆ p ( x, t ) = ϕ ⊤ X,T M ε ϕ X,T , following holds under

assumption 2 on the coefficients of the fractional FPE,

for an appropriate choice of parameters. We also know that

We know that, for given A m , ˜ p ( x, t ) = ψ ( x, t ) ⊤ ˜ P M ε ˜ Pψ ( x, t ) ⊤ where ˜ P is the projection operator defined earlier. From the results in theorem 8, we have

It is clear that some big constant Q , ˜ f x j x i ( x ; M ε , η ) ≤ Q ∥ M ε ∥ , for all i, j ∈ { 1 , · · · , q } . Similarly, ˜ f x i ( x ; M ε , η ) ≤ Q ∥ M ε ∥ for all i ∈ { 1 , · · · , q } and ˜ f t ( x ; M ε , η ) ≤ Q ∥ M ε ∥ . Hence, we have,

Now, let us compute ∥ M ε ∥ . We have

Let us choose, τ = ˜ C ′ ε -2 β -2 2 β β -2 log( 1+ R ε ) . Hence,

Now,

Then, note that ∥ p 1 / 2 ∥ L 2 ( X ) = ∥ p ∥ 1 / 2 L 1 ( X ) . Hence, using the argument in [RC21], we have

Term R comes from the fact that for each time instance p ( x, t ) is a density and t ∈ (0 < R ) . Hence,

By choosing h = cσ/s with s = max( C ′ , (1 + d +1 2( β -2) ) log 1 ε +(1+ d 2 ) log(1 + R ) + log( ˆ C ) + e ) , for some big enough ˆ C . Since s ≥ e , then log s ≥ 1 , so

Hence, we have,

Combining evrything together and by the use of triangles inequality, we have

To conclude we recall the fact that ˜ x 1 , . . . , ˜ x m is a h -covering of T , guarantees that the number of centers m in the covering satisfies

Then, since h ≥ cσ/ ( C 4 log C 5 log(1+ R ) ε ) with C 4 = 1 + d/ min(2( β -2) , 2) and C 5 = ( ˆ C ) 1 /C 4 , and since σ = min( R, 1 / √ τ ) , then R/σ = max(1 , R √ τ ) ≤ 1 + √ ˜ C ′ ε -1 / ( β -2) (log 1+ R ε ) -1 / 2 , so after final calculation, we have

for some constants C 8 , C 9 and C 10 independent of ε

A.7 Proof of Corollary 1

Proof of Corollary 1. Let p ⋆ be the probability corresponding to the solution of the Fokker-Planck equation with drift µ and ˆ p be the conditional probability corresponding to the solution of the FokkerPlanck equation with drift ˆ µ . And denote by ˆ p γ ( x, t ) the m dimensional PSD model obtained by applying the construction of Thm. 2 to the Fokker-Planck equation defined in terms of ˆ µ . Hence, from the direct application of the result in [BRS16, Theorem 1.1] since ∥ µ -ˆ µ ∥ L 2 = O ( ε ) we have,

Finally due to proposition 2, for any PSD model f there exist an algorithm that samples i.i.d. from the probability p f satisfying

From, theorem 2, we have,

Now note that for any t , the function ˆ p γ ( t, · ) is still a PSD model (see [RC21]). Then, for any t ∈ (0 , T ) , we have that there exists probabilities ˆ p t,α such that

Then, since W 1 is a distance and is bounded by the L 2 distance since we are on a bounded domain, there exists a C depending on the diameter and the volume of the domain, such that for any t

W 1 ( p ⋆ ( t, · ) , ˆ p t,α ( · )) 2 ≤ C ( ∥ p ⋆ ( t, · ) -ˆ p ⋆ ( t, · ) ∥ 2 L 2 ( X ) + ∥ ˆ p ( t, · ) -ˆ p γ ( t, · ) ∥ 2 L 2 ( X ) + W 1 (ˆ p γ , p α ) 2 ) Then, by taking the expectation over t ∈ (0 , T ) we have E t W 1 ( p ⋆ ( t, · ) , ˆ p t,α ( · )) 2 ≤ C ( ∥ p ⋆ -ˆ p ⋆ ∥ 2 L 2 ( T × X ) + ∥ ˆ p ( t, · ) -ˆ p γ ( t, · ) ∥ 2 L 2 ( T × X ) + W 1 (ˆ p γ , p α ) 2 ) = O ( ε 2 )

B Estimation of Fractional Laplacian Operator on a Probability Density

Before we go into the details of the estimation, let us first recall bochner's theorem.

Theorem 9 ([Rud17]) . A continuous kernel k ( x, y ) = k ( x -y ) on R d is positive definite if and only if k ( δ ) is the Fourier transform of a non-negative measure.

Proof of Theorem 3. In our psd model approximation, we have

We will first prove the part (i) of the result. Now, we know that, fractional laplacian operator is defined as,

Now that, k is a gaussian kernel with bandwidth η . Hence,

for some c > 0 which can be computed in closed form. Now, we know that convolution of two multivariate gaussian functions are still gaussian. Hence,

for some constant c ′ > 0 which again can be computed in the closed form. Hence, we have

Finally we have,

By inspecting the above equation carefully, we can see that after appropriate scaling,

where µ = 2 jη ( x i + x j ) and Σ = 4 ηI . One can sample from N ( µ, Σ) to estimate the fractional laplacian operator for PSD model. Now, we will prove part (ii) of the theorem statement. From the bochner's theorem, we have,

where q is the probability density. In our psd model approximation, we have

Now, we know that, fractional laplacian operator is defined as,

Hence,

Hence,

C Existence of an appropriate PSD matrix M ε (Fractional Fokker-Planck Equation)

Let us redefine the mollifier function g defined as in [RC21].

where J d/ 2 is the Bessel function of the first kind of order d/ 2 and V d = ∫ ∥ x ∥≤ 1 dx = π d/ 2 Γ( d/ 2+1) . g v ( x ) = v -d g ( x/v ) . F [ f ]( · ) denotes the fourier transform of f .

C.1 Useful Results ( Fractional Fokker Planck Equation)

Lemma 7. Consider the definition of g v ( x ) = v -d g ( x/v ) where g is defined in equation (13) . Let β > 2 , q ∈ N . Let f 1 , . . . , f q ∈ W β 2 ( R d ) ∩ L ∞ ( R d ) and the function p ⋆ = ∑ q i =1 f 2 i . Let ε ∈ (0 , 1] and let η ∈ R d ++ . Let ϕ X,T be the feature map of the Gaussian kernel with bandwidth η and let H X ⊗ H T be the associated RKHS. Then there exists M ε ∈ S + ( H X ⊗ H T ) with rank( M ε ) ≤ q , such that for the representation ˆ p ( x, t ) = ϕ ⊤ X,T M ε ϕ X,T , following holds for v > 0 and k ∈ { 1 , · · · d } ,

for some positive constants c 1 and c 2 , where C ( i ) T 1 = ∥ F [ f i ] ∥ L 1 ( R d +1 ) + ∥ F [ f i ] ∥ L ∞ ( R d +1 ) ∥ g ∥ L 1 ( R d +1 and C ( i ) T 2 = ∥ ∥ ∥ ω ∥ 2 s F ( f i )( ω ) ∥ ∥ L 1 ( R d +1 ) + ∥ ∥ ∥ ω ∥ 2 s F [ f i ]( ω ) ∥ L 1 ( R d +1 ) ∥ g ∥ ∥ L 1 ( R d +1 ) for i ∈ { 1 , · · · , q } .

Proof. As in [RC21], let us denote, f i ⋆ g v as f i,v . We consider

Let us consider the function,

Hence,

Now, we can apply Young's convolution inequality on T 1 and T 2 . Hence,

In the last equation, we applied holder's inequality. From uniform continuity and the Riemann-Lebesgue lemma, we have

Hence,

Similarly,

In the above equation, we can apply holder's inequality to get,

Hence,

The term ∥ m ( ω ) ∥ L 2 ( R d +1 ) and ∥∥ ω ∥ 2 s m ( ω ) ∥ L 2 ( R d +1 ) can be bounded using similar technique as before. We will use the result from Lemma 1.

Combining everything together, we have

C.2 Proof of Theorem 4

From the previous lemma, we have,

for some positive c 1 and c 2 where C ( i ) T 1 = ∥ F [ f i ] ∥ L 1 ( R d +1 ) + ∥ F [ f i ] ∥ L ∞ ( R d +1 ) ∥ g ∥ L 1 ( R d +1 ) and C ( i ) T 2 = ∥ ∥ ∥ ω ∥ 2 s F ( f i )( ω ) ∥ ∥ L 1 ( R d +1 ) + ∥ ∥ ∥ ω ∥ 2 s F [ f i ]( ω ) ∥ L 1 ( R d +1 ) ∥ g ∥ ∥ L 1 ( R d +1 ) . Now, if we choose, for some constant c

then

We can bound the other two terms as using lemma 2. We have,

If we choose where C 1 = ∑ i =1 ∥ f i ∥ W β 2 ( R d +1 ) ∥ f i ∥ L ∞ ( R d +1 ) and C 2 ∑ q i =1 ∥ f i ∥ W β 2 ( R d ) ∥ ∥ ∥ ∂f i ∂t ∥ ∥ ∥ L ∞ ( R d +1 ) ∥ g ∥ L 1 ( R d +1 ) . Hence,

Similarly, from lemma 2

From [RC21, Theorem D.4 (Eq. D.27) ], we have,

Hence, if we choose,

where C 1 = ∑ i =1 ∥ f i ∥ W β 2 ( R d +1 ) ∥ f i ∥ L ∞ ( R d +1 ) , C 2 = ∑ q i =1 ∥ f i ∥ W β 2 ( R d +1 ) ∥ ∥ ∥ ∂f i ∂t ∥ ∥ ∥ L ∞ ( R d +1 ) ∥ g ∥ L 1 ( R d +1 ) and C 3 = (1 ∥ g ∥ L 1 ( R d +1 ) ) ∑ q i =1 ∥ f i ∥ W β 2 ( R d +1 ) ∥ f i ∥ L ∞ ( R d +1 ) , then, we have

Hence, by combining everything together, we have

We can bound the trace of the matrix as follows. We have,

From [RC21, Equation D.21], we have

Since, we choose,

where C depends on f i s, F [ f i ] and it's derivative.

C.3 Approximation Properties of PSD Model for Fractional Laplacian (Proof of Theorem 4)

Proof. From the previous result in theorem 4, we know that there exists an M ε ∈ S + ( H X ⊗ H T ) with rank( M ε ) ≤ q , such that for the representation ˆ p ( x, t ) = ϕ ⊤ X,T M ε ϕ X,T , following holds under assumption 2 on the coefficients of the fractional FPE,

for an appropriate choice of parameters. We also know that

We know that, for given A m , ˜ p ( x, t ) = ψ ( x, t ) ⊤ ˜ P M ε ˜ Pψ ( x, t ) ⊤ where ˜ P is the projection operator defined earlier. We will utilize the Gagliardo-Nirenberg inequality to control the error in compression of fractional laplacian compressed model with respect to the original one. For any fixed t ,

By applying Gagliardo-Nirenberg inequality for fractional laplacian [MP18], we have

where ϑ = s ≤ 1 and ∆ is normal laplacian operator. Let us consider the term T 1 and T 2 separately. In our case, β t ( x ) = ψ ( x, t ) ⊤ ( ˜ P M ε ˜ P -M ε ) ψ ( x, t ) . In the proof of theorem 8, we have proved the bound on ∆ β t ( x ) and β t ( x ) . From equation (43), we have,

where c M = ∥ M ε ∥ and we denoted by ˜ u 1 ( x ) the quantity ˜ u 1 ( x ) = ∥ ( I -˜ P ) ψ ( x, t ) ∥ H η and we noted that ∥ M 1 / 2 ε ψ ( x, t ) ∥ 2 H η = ψ ( x, t ) ⊤ M ε ψ ( x, t ) = f ( x ; M ε , ψ ( x, t )) . Similarly, we have shown that

From equations (46) and (47)

where ˜ f x j x i ( x ; M ε , ψ ) = ∂ 2 ψ ( x,t ) ∂x j x i ⊤ M ε ∂ 2 ψ ( x,t ) ∂x j x i and ˜ u x j x i ( x ) = ∥ ∥ ∥ ( I -˜ P ) ∂ 2 ψ ( x,t ) ∂x j x i ∥ ∥ ∥ H η . And,

Follwoing similar steps of calculations as that is Theorem 8 and Theroem 2, we get

for some constant P 1 , P 2 and P 3 . Now, let us compute ∥ M ε ∥ . We have

Let us choose, τ = ˜ C ′ ε -2 β -2 s 2 β β -2 s log( 1+ R ε ) . Hence,

For some constant C 3 . Hence, Then, note that ∥ p 1 / 2 ∥ L 2 ( X ) = ∥ p ∥ 1 / 2 L 1 ( X ) . Hence, using the argument in [RC21], we have

Term R comes from the fact that for each time instance p ( x, t ) is a density and t ∈ (0 < R ) . Now,

By choosing h = cσ/s with s = max( C ′ , (1 + d +1 2( β -2 s ) ) log 1 ε +(1+ d 2 ) log(1 + R ) +log( ˆ C ) + e ) , for some big enough ˆ C . Since s ≥ e , then log s ≥ 1 , so

Hence, combining evrything, we have from triangles inequality,

To conclude we recall the fact that ˜ x 1 , . . . , ˜ x m is a h -covering of T , guarantees that the number of centers m in the covering satisfies

Then, since h ≥ cσ/ ( C 4 log C 5 log(1+ R ) ε ) with C 4 = 1 + d/ min(2( β -2 s ) , 2) and C 5 = ( ˆ C ) 1 /C 4 , and since σ = min( R, 1 / √ τ ) , then R/σ = max(1 , R √ τ ) ≤ 1 + √ ˜ C ′ ε -1 / ( β -2) (log 1+ R ε ) -1 / 2 , so after final calculation, we have

for some constants C 8 , C 9 and C 10 independent of ε

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article investigates the approximation of the Fokker-Planck equation and the fractional Fokker-Planck equation using positive semi-definite (PSD) models. The authors demonstrate that for a given accuracy $\\epsilon$, there exists a PSD model with a dimension not exceeding $\\epsilon^{-(d+1)/(\\beta - 2s)}(\\log 1/\\epsilon)^{d+1}$ that approximates the corresponding partial differential equation (PDE). By utilizing this result in conjunction with the PSD model-based sampling method proposed in [1], the authors are able to provide samples of the stochastic differential equation (SDE) corresponding to the Fokker-Planck equation or fractional Fokker-Planck equation at any given time $t$.\n\n[1]Marteau-Ferey U, Bach F, Rudi A. Sampling from arbitrary functions via psd models[C]//International Conference on Artificial Intelligence and Statistics. PMLR, 2022: 2823-2861.\n\nSTRENGTHS:\n1. This article demonstrates clear and logical writing, making it easy to understand the problem formulation considered by the authors as well as the fundamental concepts and properties of the PSD model adopted.\n2. As a newly proposed model, this paper presents a novel application scenario for the PSD model, which has not been previously explored in related works. This contributes to its significant novelty.\n3. From the perspective of sampling based on SDEs, the authors provide a new design approach for sampling by combining existing works, offering a fresh perspective on sampling techniques.\n\nWEAKNESSES:\n1. Although the author mentions it in the paper, the influence of initial conditions on the solution of the PDE can sometimes be significant. It would enhance the completeness of the theoretical analysis in this article if the author could provide an analysis of whether the PSD model can satisfy the initial conditions or, in cases where exact satisfaction is not possible, characterize the approximation of the initial conditions' impact on the final solution's approximation.\n2. Personally, I have limited knowledge of non-log-concave sampling and numerical modeling of PDEs. However, it would be beneficial if the author could compare the effectiveness of the PSD model-based sampling method with other traditional methods (such as Langevin Monte Carlo) in terms of sampling performance. Additionally, comparing the approximation methods of the PSD model for PDE solutions with other existing approaches, either from the perspective of complexity analysis or purely experimental results, would provide valuable insights.\n\nQUESTIONS:\n1. In line 180, should it be $E e^{j\\langle u, L_{t-s}^\\alpha\\rangle} = \\exp\\{-(t-s)||u||_2^\\alpha\\}$?\n2. In line 216, should it be $Y \\in \\mathbb{R}^{n\\times d^\\prime}$?\n\nLIMITATIONS:\nDespite the theoretical guarantees provided by the author, I believe the biggest limitation of this article lies in the lack of clear demonstration of the advantages of the PSD model. It would be beneficial if the author could explain how the PSD model stands out compared to other similar models in terms of approximating FPE and subsequent sampling tasks. This would significantly enhance the persuasiveness of the article.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduced a new method of sampling solutions of the Fokker--Planck equation (FPE) using a positive semidefinite (PSD) model. While the algorithm used for the PSD model is not new, the authors created a framework to sample from the FPE, which is quite original based on my knowledge. \n\nWhile I believe the results are very neat, I would like to clarify several questions with the authors before providing a confident score. For now I will recommend borderline accept, and will raise the score once my questions are adequately addressed.\n\nSTRENGTHS:\nThe approach to drawing samples for the Fokker--Planck is very unique and original. I believe alternative methods that hold promise should be welcomed and more carefully studied, regardless of how practical it is at the moment.\n\nWEAKNESSES:\nThere are a few clarifications I would like to have regarding the implementation of the algorithm.\n\nQUESTIONS:\n1. To start, while I understand the algorithm of [MFBR22] is not part of the contribution for this work, can the authors provide a high level explanation of how the algorithm works, and comment on its computational complexity? I believe all readers can benefit from understanding the core algorithm, and plus I believe readers should be able to implement the algorithm presented in this paper without having to reference other works. \n\n2. Can authors clarify the role that $q$ plays in this problem? In particular, is $q$ given or chosen by the user, and how does the value of $q$ affect the computational complexity of this algorithm? \n\n3. Under the discussion of \"Efficient sampling method\" the authors mentioned a step of finding the best model $\\hat{A}_m$ by minimizing an error function. Suppose I am an applied scientist who wants to sample from a given FPE, how would I compute this model $\\hat{A}_m$ from the PDE? On a high level, I would like to understand how to implement this sampler end to end, and I don't think I fully understand this part. \n\nOnce again, I would be happy to raise my score once my questions are adequately answered.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers numerical methods for the Fokker-Plank equation and its \"fractional Laplacian\" version. These partial differential equations describe the evolution of densities of certain random processes: in the case of standard FP, the process is a regular diffusion (an SDE driven by Brownian motion), whereas the fractional version corresponds to a process driven by $\\alpha$-stable noise. Accordingly, the authors focus on initial conditions given by probability densities. \n\nThe idea behind the paper is to adapt the fairly recent methodology of positive semidefinite models (PSD) to this PDE problem. PSD models estimate nonnegative functions via feature maps $\\phi:X\\to H$, where $X$ is the domain of the function and $H$ is a Hilbert space. Given $\\phi$, the nonnegative function of interest is estimated as $\\langle \\phi, M\\phi\\rangle$, where the PSD linear operator $M$ is chosen from the data. The whole method is based on the same idea as the well-known kernel trick: there is no need to explicitly deal with the Hilbert space $H$; only the kernel matrix over data points will matter.  \n\nIn a nutshell, the paper gives a method for approximately solving the two aforementioned PDEs via PSD models. The salient features of this approach are as follows. \n\n1) The PSD estimator can be obtained via a semidefinite program (as shown in previous work). \n\n2) The method will output probability densities, and one can sample from them via methods from previous work. \n\n3) The complexity of the problem grows like $\\approx \\epsilon^{-d/\\beta}$, where $\\epsilon$ is the target accuracy, $d$ is the dimension and $\\beta$ measures the degree of smoothness of the solution. The exponent is not quite right, but the point is that sufficient smoothness will avoid the curse of dimensionality.\n\nOne may note that this method has rigorous error guarantees. This sets it apart from other PDE solvers based on machine learning ideas, such as Physics Inspired Neural Nets (PINNs).\n\nSTRENGTHS:\nThe paper gives strong convergence guarantees for a new PDE solving method. Although several ingredients come from previous work, they are combined with new ideas into an impressive package. As mentioned above, this is a ML-based PDE solver with rigorous guarantees. Importantly, these guarantees do not require coercivity or hypercontractivity properties that are usually required from Langevin samples for solutions.\n\nWEAKNESSES:\n* No experiments are performed to compare the method with other PDE solvers (though this is understandable in a heavily mathematical paper). \n* The writing is a bit sloppy in places: I found quite a few typos and have several minor comments on writing (I could probably find some more typos if I kept looking). See the list below. \n\nNB: were it not for the sloppiness, I would have given this paper a higher score.\n\n---\n\n_List of typos and comments on writing_\n\n(Line numbers refer to the full paper in the SM.)\n\nLines 146/7: \"It was\" and \"It has\" should be \"They were\" and \"They have\".\n\nLine 180: why define $j=\\sqrt{-1}$ if you could have just written $\\sqrt{-1}$ in the exponent? (You never use this notation again, do you?)\n\nLine 185: Shouldn't the condition for infinite $p$-th moment be $p\\geq \\alpha$?\n\nLine 201 and elsewhere: I find it a bit odd that the transpose notation is used for vectors in Hilbert space. If you are going to use it, at least explain here what is going on.\n\n \nLine 251: why is this sensence needed here?\n\nLine 205: $\\eta$ was treated as a vector before, and will be treated as a vector later in the text. Here, however, it seems to be a scalar. \n\nLine 223: \"The cost of the algorithm is...\" -- per sample point, right?\n\nLine 254: \"Hence, $p^\\star(x,t)$\"\n\nLine 255 should end with \":\".\n \nLine 270: the definition of $\\psi$ should be in a numbered display for easier refererence.\n\nDisplay (8): if $p^*$ is the true solution, why do we have it in here? (Its contribtion vanishes.) Indeed, the definition of the loss later does not involve $p^*$; see eg. (9). (On the other hand, $p^\\star$ shows up again in the display right after line 989.)\n\nLine 294 and elsewhere: when you are mentioned a numbered theorem or assumption, start the corresponding word with a capital letter. \n\nLine 295: $\\mathcal{H}_X\\otimes\\mathcal{H}_T$\n\nLine 318: \"by Gaussian kernel approximation\" -- should this be here?\n\nLine 365: \"Similarly to the previous section, two steps are required to obtain\"\n\nLine 327 and elsewhere: I think $\\mathbb{E}_t$ was not defined anywhere.\n\nLine 352: \"If the kernel satisfies the Bochner...\"\n\nLine 355 and elsewhere: this is a matter of taste, but couldn't you replace \"utilize\" with \"use\". \n\nLine 376: \"The major difficulty\"\n\nLine 391 (taste): \"does require\" could be \"require\".\n\nLine 395: I think \"approximated\" should be \"approximating\".\n\nProposition 4 is also in Evans's book, which was cited previously (I am saying this from memory, so please check. $W_2$ is replaced by $H$ in that book.)\n\nMany spots of the SM: the inequality $\\|f\\star g\\|_p\\leq \\|f\\|_p\\|g\\|_1$ should be mentioned, as it is used in several steps. \n\nLemma 2: it seems that $\\tilde{p}$ is replaced by $f$ in the statement of the Lemma and at several steps of the proof. (Eg. the display right after 665.) This also seems to take place in the next Lemma.\n\nLine 680: Since you are repeating a definition from the preceding proof, please say so. \n\nLine 682: \"We use the result of Lemma 1\"\n\nLine 746: Should \"The following result holds\" be here?\n\nDisplay (34): this \"fill distance\" is the smallest value of $h$ such that $\\tilde{X}$ is an $h$-net of $[-R,R]^d$.\n\nDisplay below line 777: how should I interpret the norm in the RHS?\n\nLines 873/4: why are you spaeking of conditional probabilities here?\n\nLine 891: what is $f(\\delta)$?\n\nLine 910: \"redefine\" should be \"define again\" (they are not the same).\n\nLine 928: capitalize \"Holder\".\n\nQUESTIONS:\n* Have you attempted experiments against other ML approaches, or other methods from the literature?\n* Methods have been proposed recently to speed up SDPs, including randomized sketches  (https://math.paperswithcode.com/paper/scalable-semidefinite-programming). Could these methods be applicable in the present setting?\n\nLIMITATIONS:\nThey have not addressed limitations explicitly, but I do not think that that would have been necessary.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper studies the problem of efficiently sampling from a SDE given a drift function and diffusion tensor. It proposes a solution based on computing a PSD model that satisfies the Fokker-Planck solution associated with the SDE, and then sampling from the resulting PSD model.

STRENGTHS:
The paper seems to have a lot of technical content, and it is clear that the authors have put in a lot of effort.

WEAKNESSES:
This paper is extremely difficult to read. The theorem statements (and the paper overall) need to be drastically simplified to be understandable. The problem doesn't seem to be well motivated, and it's not clear from reading the paper how the solution compares to others in the literature. For instance, the [LCL+21] paper is mentioned in related work, but it's not at all clear how the solution proposed of learning a PSD model compares to their work. There are no experiments to support this idea of fitting a PSD model. Overall, I think this paper needs to be rewritten to be more easily understandable.

QUESTIONS:
- Can you provide experimental evidence for the theoretical claims?
- Can you provide simplified theorem statements?
- Can you provide a more clear comparison with related work?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-07