Title: An Efficient High-dimensional Gradient Estimator for Stochastic Differential Equations
Venue: NeurIPS 2024
Accepted: yes
Keywords: derivative estimation

gradient estimation

sensitivity analysis

stochastic differential equations

jump diffusions

neural SDEs
Abstract: Overparameterized stochastic differential equation (SDE) models have achieved remarkable success in various complex environments, such as PDE-constrained optimization, stochastic control and reinforcement learning, financial engineering, and neural SDEs. These models often feature system evolution coefficients that are parameterized by a high-dimensional vector $\theta \in \mathbb{R}^n$, aiming to optimize expectations of the SDE, such as a value function, through stochastic gradient ascent. Consequently, designing efficient gradient estimators for which the computational complexity scales well with $n$ is of significant interest. This paper introduces a novel unbiased stochastic gradient estimator—the generator gradient estimator—for which the computation time remains stable in $n$. In addition to establishing the validity of our methodology for general SDEs with jumps, we also perform numerical experiments that test our estimator in linear-quadratic control problems parameterized by high-dimensional neural networks. The results show a significant improvement in efficiency compared to the widely used pathwise differentiation method: Our estimator achieves near-constant computation times, increasingly outperforms its counterpart as $n$ increases, and does so without compromising estimation variance. These empirical findings highlight the potential of our proposed methodology for optimizing SDEs in contemporary applications.

An Efficient High-Dimensional Gradient Estimator for Stochastic Differential Equations

Abstract

Overparameterized stochastic differential equation (SDE) models have achieved remarkable success in various complex environments, such as PDE-constrained optimization, stochastic control and reinforcement learning, financial engineering, and neural SDEs. These models often feature system evolution coefficients that are parameterized by a high-dimensional vector θ ∈ R n , aiming to optimize expectations of the SDE, such as a value function, through stochastic gradient ascent. Consequently, designing efficient gradient estimators for which the computational complexity scales well with n is of significant interest. This paper introduces a novel unbiased stochastic gradient estimator-the generator gradient estimator-for which the computation time remains stable in n . In addition to establishing the validity of our methodology for general SDEs with jumps, we also perform numerical experiments that test our estimator in linear-quadratic control problems parameterized by high-dimensional neural networks. The results show a significant improvement in efficiency compared to the widely used pathwise differentiation method: Our estimator achieves near-constant computation times, increasingly outperforms its counterpart as n increases, and does so without compromising estimation variance. These empirical findings highlight the potential of our proposed methodology for optimizing SDEs in contemporary applications.

1 Introduction

Weconsider a family of jump diffusions { X x θ ( t, s ) ∈ R d : s ∈ [ t, T ] } that are generated by stochastic differential equations (SDEs) and indexed by the initial condition x ∈ R d at time s and a parameter θ ∈ Θ ⊂ R n . In modern applications, the parameter θ , encoding characteristics of an engineering model, often represents the weights of a deep neural network. This paper focuses particularly on scenarios where the dimension n of θ is significantly greater than the dimension d of the space. This setting naturally arises in the implementation of large AI architectures in modern applications.

Concretely, for each 1 ≤ i ≤ d , the i 'th entry of X x θ ( t, · ) , denoted by X x θ,i ( t, · ) , satisfies the Itô SDE:

Here, { µ θ,i : 1 ≤ i ≤ d } and { σ θ,i,k : 1 ≤ i, k ≤ d } are the drift and volatility, respectively, satisfying suitable regularity conditions (to be discussed). For simplicity in our introductory explanations, we will assume that the jump term J θ is zero. However, incorporating this jump feature is valuable in many applied settings, and arises in various fields such as financial engineering [17], stochastic control [6], and neural SDE models [7]. Accordingly, we will fully integrate and discuss the jump components in our main results in Section 3.

The primary objective of this paper is to develop an efficient gradient estimator, with respect to θ , for a large class of path-dependent expectations derived from an SDE. Concretely, we consider

The value v θ ( t, x ) represents the expected cumulative reward running X x θ from time t to T . Here, ρ θ and g θ represents the reward rate and the terminal reward, respectively. This formulation encompasses a wide range of science and engineering problems including PDE-constrained optimization [22, 20], stochastic control and reinforcement learning [8], and neural SDE models [23].

The gradient ∇ θ v θ ( t, x ) = ( ∂ θ 1 v θ ( t, x ) , . . . , ∂ θ n v θ ( t, x )) ∈ R n is of significant interest in the sensitivity analysis, learning, and optimization of these models. In particular, finding an efficient unbiased estimator for ∇ θ v θ ( t, x ) with low variance is essential if one is to apply stochastic gradient descent to find near optimal policies or model parameters within the parametric class θ ∈ Θ .

Under reasonable smoothness and integrability conditions, it is natural to consider the pathwise differentiation estimator obtained by applying infinitesimal perturbation analysis (IPA) to the sample path of X x θ w.r.t. the i th coordinate of θ . For instance, if ρ θ ( · ) = ρ ( · ) independent of θ and g = 0 , then we have a representation

where ∂ θ i X x θ ( t ) is the pathwise derivative of the process X x θ w.r.t. θ i . The processes { X x θ,j , ∂ θ i X x θ,j : i = 1 , . . . , n ; j = 1 , . . . , d } satisfy a system of d + d · n SDEs [11, Equation (3.31)], which must be jointly simulated. Therefore, to estimate the gradient, the pathwise differentiation method requires simulating this d + d · n dimensional SDE. Note that the dimension is linear in n , the dimension of the parameter space. Contemporary applications of SDEs in physics-informed and data-driven environments such as deep neural SDEs and deep RL where overparameterization excel, necessitate a model with exceptionally large n that is often many orders of magnitude larger than d . Hence, simulating the SDE of dimension d + d · n becomes extremely resource-intensive. Motivated by these applications, we ask the following question:

Can we device an efficient, unbiased, and finite variance estimator for ∇ θ v θ ( t, x ) with a computation time insensitive to n ?

The answer is affirmative. Precisely, our main contribution is designing the unbiased generator gradient estimator of ∇ θ v θ ( t, x ) that requires only simulating O ( d 2 ) SDEs when the volatility parameters σ θ do not depend on θ and O ( d 3 ) SDEs in the general setting, as summarized in Table 1.

We remark that in addition to pathwise differentiation, likelihood ratio-based estimators are also popular for sensitivity analysis in SDEs; see e.g. Yang and Kushner [26]. However, typically they are only applicable if σ θ is independent of θ and under more restrictive jump structures. When applicable, likelihood ratio-based estimators could be appealing alternatives as they introduce a change of measure that represents the derivatives as a functional of the d -dimensional processes X x θ . Nevertheless, these estimators typically have significantly higher variance.

Finally, we apply our estimator to linear-quadratic control problems and test its performance in optimizing neural-network-parameterized controls. As we increase the number of network parameters n , the results in Figure 1a and Table 2 highlight a substantial improvement in computational efficiency, as compared to the pathwise differentiation method, while still maintaining competitive variance levels. Furthermore, Figure 1a confirms that the computation time of our estimator is robust to increases in n , even in extremely high-dimensional scenarios with n approaching 10 8 .

1.1 Literature Review

Gradient Estimation: Gradient estimation, particularly likelihood ratios and IPA methods, is crucial in sensitivity analysis. Foundational works in the late 20th century by Glynn [5, 4] and further adaptations to the SDE setting [26, 3] highlight these developments. IPA has evolved to apply stochastic flow techniques to SDEs, both with and without reflecting boundaries [23, 12, 16, 24, 14].

Applications of Gradient Estimators: Gradient estimators are widely used in stochastic control and reinforcement learning (RL) models. Policy gradient methods in discrete-time RL, including REINFORCE and deep policy gradient approaches, are notable applications [25, 13, 21]. Continuoustime RL have been explored using policy gradients in settings with continuous diffusion dynamics [8]. Jump diffusions are important models in financial engineering and stochastic control [17, 15, 9, 6]. Gradient estimators can also be used for optimizing these models. Neural SDE models are modern computational frameworks that model the dynamics of stochastic systems using a neural-networkparameterized SDE. Chen et al. [1], Tzen and Raginsky [23], Kidger [10] focus on the continuous case, while Jia and Benson [7] consider ODEs modulated by compound Poisson jumps. Efficient gradient estimators in high-dimensional settings are crucial for fitting these SDE models.

Diffusion with Jumps and Stochastic Flow: The main technical tools for this paper are SDEs with jumps and stochastic flows. Our references are Protter [19], Kunita [11], Øksendal and Sulem [18].

1.2 Remarks on Paper Organizations

The paper is structured as follows: Section 2 outlines the core concepts of our estimator in a zero-jump setting, focusing on intuitive understanding over technical detail. In Section 3, we introduce the SDE model with jumps and provide a set of sufficient conditions that rigorously support the earlier insights. While more general and complex assumptions exist that lead to similar conclusions, these are presented in Appendix A to align with the concise format of the conference proceedings. The paper concludes with Section 4, where we conduct numerical experiments on neural-network-parameterized linear-quadratic control problems, demonstrating the effectiveness of our methodology.

2 Key Methodological Insights

In this section, we motivate our proposed generator gradient estimator by first providing a nonrigorous derivation. We assume the SDE model (1.1) where the jumps J θ ≡ 0 and Θ ⊂ R n is a bounded open neighbourhood of the origin. W.l.o.g, we are interested in estimating the gradient at θ = 0 ∈ Θ and t = 0 ; i.e. ∇ θ v 0 (0 , x ) = ∇ θ v θ (0 , x ) | θ =0 .

To simplify notation, we denote X x θ ( t ) := X x θ (0 , t ) and X x θ ( t -) := X x θ (0 , t -) , and the function

Also, for function v θ ( t, x ) , we use ∂ i v θ ( t, x ) to denote the space derivative ∂ i v ∂x ∣ ∣ θ,t,x and ∇ the space gradient. Similarly, ∂ θ i and ∇ θ denotes the θ partials.

Under sufficient regularity conditions, by the Feynman-Kac formula, v θ in (1.2) is the solution to the partial differential equation (PDE)

for all θ ∈ Θ , where L θ is the generator of X x θ given by

for f that is twice differentiable in x . Assuming enough smoothness, we formally differentiate the PDE (2.2) w.r.t. θ i and then set θ = 0 to obtain

Here, the operator ∂ θ i L 0 is defined as

)

Interpreted as the derivative of L θ w.r.t. θ at 0 , this inspires the name "generator gradient" method. Next, define u 0 = ∂ θ i v 0 . Treating ∂ θ i L 0 v 0 as fixed, we observe that u 0 solves the PDE (2.3) which is of the form (2.2). Hence, applying the Feynman-Kac formula again to ∂ θ i v 0 (0 , x ) = u 0 (0 , x yields the following expectation representation

Note that the expression inside the expectation contains only space derivatives (due to ∂ θ i L 0 ) of the value function v 0 but not the θ derivatives. In particular, if we can estimate the gradient ∇ v 0 ( t, x ) and the Hessian matrix H [ v 0 ]( t, x ) := { ∂ i ∂ j v 0 ( t, x ) : 1 ≤ i, j ≤ d } efficiently, then the representation in (2.5) will lead to a natural estimator of ∂ θ i v 0 (0 , x ) .

To estimate ∇ v 0 ( t, x ) and H [ v 0 ]( t, x ) , we employ the pathwise differentiation estimator from (1.3). Specifically, under enough regularity conditions, we can interchange the derivatives and integration

Here, we write ∇ X x 0 := { ∂ a X x 0 ,i : i, a = 1 , . . . d } and H [ X x 0 ] := { ∂ b ∂ a X x 0 ,i : i, a, b = 1 , . . . d } . The notation 〈 ∇ h, H [ X x 0 , · ] 〉 := ∑ d a =1 ∂ a hH [ X x 0 ,a ] ∈ R d × d for h = ρ 0 , g 0 . The dependence of ρ 0 , g 0 on time and the state process is hidden.

We estimate these expectations by simulating the SDEs for { X x 0 , ∇ X x 0 , H [ X x 0 ] } given by (1.1) and

where the dependence of the coefficients on r , X x 0 ( t, r -) , and z , as well as the dependence of X x 0 , ∂ a X x 0 , ∂ a ∂ b X x 0 on ( t, s ) , ( t, r -) are suppressed.

The dimension of these SDEs is d + d 2 + 1 2 d 3 , where the 1 2 comes from the Hessian being symmetric. Moreover, when the volatility σ is independent of θ , our method only necessitates estimating ∇ v 0 . This reduction leads to simulating the SDEs for { X x 0 , ∇ X x 0 } of dimension only d + d 2 .

Assuming sufficient integrability, the unbiasedness of Z implies

which we will elaborate upon in (A.1). The same holds for the H ( t, x ) process as well. Therefore, we can replace the derivatives ∇ v 0 with Z and H [ v 0 ] with H in (2.5) without changing the expectation.

Also note that producing a sample of Z ( t, x ) requires simulating the solution to SDEs (1.1) and (2.7) within time [ t, T ] starting from x, I, 0 . So, it is not very efficient to compute Z ( t, X x 0 (0 , t )) for every t ; a similar issue exists for H as well. This can be addressed by randomizing the integral.

With these considerations, we proceed to define the generator gradient estimator. First, let ∇ θ L 0 V 0 ( t, x ) be defined by replacing ∂ i v ( t, x ) with Z i ( t, x ) and ∂ j ∂ i v with H i,j ( t, x ) in the definition (2.4) of ∂ θ i L 0 v 0 ( t, x ) . Then, define the generator gradient estimator as

where τ ∼ Unif[0 , T ] is sampled independently. We can also randomize the integral of ∇ θ ρ 0 ( t, X x θ ( t )) if the gradient is hard to compute. With the derivation in (2.8), it is easy to see that ED ( x ) = ∇ θ v 0 (0 , x ) is unbiased.

In summary, due to the observation in (2.5), we are able to "move" the estimation of ∇ θ v 0 onto that of ∇ v 0 and H [ v 0 ] . This results in a significant reduction in the dimension of the SDEs we need to simulate, underlying the remarkable efficiency of our methodology, especially when the dimension n of θ significantly exceeds d .

3 Jump Diffusions and the Generator Gradient Estimator

In this section, we rigorously formulate a jump diffusion process driven by an SDE. We extend the generator gradient estimator to this context by first rigorously establishing an expectation representation of the derivative as in (2.5). Then, we also validate the representation (2.6) using the jump version of (2.7). These lead to our generator gradient estimator in the jump diffusion context. To improve the clarity of the paper (at a cost of generalizability), we will state a set of sufficient assumptions that are easy to verify. However, we will state and prove our theorems using a set of more general assumptions in the Appendix A.

We consider jump diffusions on the canonical probability space of càdlàg functions [0 , T ] → R d generated by SDEs of the form (1.1) where the jump term is given by

In this expression, B is a standard Brownian motion in R d ′ ; ˜ N is a compensated Poisson random measure with intensity measure dt × ν ( dz ) with ν a Lévy measure on ( R d ′ 0 := R d ′ \ { 0 } , B ( R d ′ 0 )) , i.e. ∫ R d ′ 0 1 ∧ | z | 2 ν ( dz ) < ∞ ; the -r notation in X x θ ( t, r -) denotes the left limit; and the stochastic integrations are Itô integrals. Here, for a vector/matrix/tensor v ∈ R d 1 × d 2 × d 3 , we denote | v | 2 := ∑ i,j,k | v i,j,k | 2 . We further define γ ( z ) = | z | ∧ 1 and µ ( dz ) = γ ( z ) 2 ν ( dz ) . Then µ is a finite measure on ( R d ′ 0 , B ( R d ′ 0 )) . Also, since we are interested in the gradient at θ = 0 , we can assume w.l.o.g. that Θ is a bounded open neighbourhood of 0 .

The generator of this system of SDEs is L θ := L C θ + L J θ , where

for f ∈ C 1 , 2 ([0 , T ] , R d ) . We remark that for open subsets W , X , the space C i,j,k ([0 , T ] , W , X ) represents the set of functions f on [0 , T ] × W × X that has continuous mixed partial derivatives ∂ a t ∂ b w ∂ c x f on (0 , T ) × W × X for every a ≤ i, b ≤ j, c ≤ k . Moreover, these mixed partial derivatives have continuous extensions on [0 , T ] × W × X .

3.1 Probabilistic Representation of the Gradient

In this section, we rigorously establish the probabilistic representation of the gradient ∇ θ v 0 (0 , x ) as outlined in equation (2.5). Our approach leverages the continuous dependence of θ → X x θ of the solutions to (3.1) in a neighbourhood of 0 , given sufficient regularity conditions. This behavior extends the properties associated with stochastic flows, as explained in the work by Kunita [11].

Recall that Θ is a bounded neighbourhood of 0 ∈ R n . To clarify the assumptions, we enlarge Θ and consider Θ ϵ = { θ + v : θ ∈ Θ , v ∈ B n (0 , ϵ ) } where B n (0 , ϵ ) is the open ball in R n at 0 of radius ϵ .

Assumption 1. For some ϵ > 0 , the following regularity conditions hold

The mappings ( s, θ, x ) → µ θ ( s, x ) , σ θ ( s, x ) , ρ θ ( s, x ) , g θ ( s, x ) are C 0 , 1 , 1 ([0 , T ] , Θ ϵ , R d ) . For each z ∈ R d ′ 0 , ( s, θ, x ) → χ θ ( s, x, z ) /γ ( z ) is C 0 , 1 , 1 ([0 , T ] , Θ ϵ , R d ) . Moreover, | χ θ ( s, 0 , z ) /γ ( z ) | is uniformly bounded in s ∈ [0 , T ] and z ∈ R d ′ 0 .

The spacial derivatives |∇ µ θ | , |∇ σ θ | , and |∇ χ θ | are uniformly bounded. The θ derivatives satisfy linear growth

for all s ∈ [0 , T ] , x ∈ R d , z ∈ R d ′ , and θ ∈ Θ .

The θ derivatives of the rewards satisfy polynomial growth: for some m ≥ 1 ,

Remark. Requirement 1 implies that for each fixed x , the θ derivatives of the coefficients are uniformly bounded in [0 , T ] × Θ , as Θ is assumed to be bounded. So, the seemingly strong requirements of the θ derivative satisfying the growth condition in items 2 and 3 are not very restrictive. The boundedness of χ θ ( s, x, z ) /γ ( z ) in z is relaxed in Assumption 5 in the appendix, allowing unbounded jumps. The strong condition is the uniform boundedness of |∇ µ θ | , |∇ σ θ | , and |∇ χ θ | . However, this is typically necessary for the existence and uniqueness of strong solutions to the SDE (3.1).

Assumption 2. Assume that { v θ ∈ C 1 , 2 ([0 , T ] , R d ) : θ ∈ Θ } are classical solutions to the partialintegro-differential equations (PIDE)

where L θ = L C θ + L J θ are defined in (3.2) . Moreover, v θ and its space derivatives satisfy polynomial growth: for each θ ∈ Θ , there exists 0 < c θ < ∞ and m ≥ 1 s.t.

Remark. By classical solution, we mean that v θ satisfies ∂ t v θ + L θ v θ + ρ θ = 0 on (0 , T ) × R d with its continuous extensions of satisfying v θ ( T, · ) = g θ . This is possible, for example, in settings with C 2 terminal rewards. Note that is a stronger requirement compared to the definition in Evans [2].

As we have motivated in Section 2, Assumption 2 follows from a generalized version of the FeynmanKac formula, under additional technical assumptions. Moreover, the growth of v θ and its space derivatives can be derived from assumptions on the growth of the rewards. However, in order to not obscure the main message of the paper and to streamline the proof, we directly assume these properties. We refer interested readers to Kunita [11, Chapter 4] where stochastic flow techniques similar to the proofs in the paper are employed to establish the PIDE and validate the growth rates.

Theorem 1 (Probabilistic Representation of the Gradient) . If Assumptions 1 and 2 are in force, then θ → v θ (0 , x ) is differentiable at 0 . Moreover, the gradient

where ∇ θ L 0 := ∇ θ L C 0 + ∇ θ L J 0 s.t. for f ( t, x ) ∈ C 1 , 2 ,

In Theorem 1, we have successfully established an expectation representation of the gradient ∇ θ v 0 (0 , x ) of the form (2.5). This naturally leads to the consideration of using Monte Carlo to estimate ∇ θ v 0 (0 , x ) . However, one observes that the representation in Theorem 1 involves the space derivatives ∂ i v 0 ( t, x ) and ∂ i ∂ j v 0 ( t, x ) , which are usually hard to compute exactly.

In the next section section, following the heuristics in (2.6) we establish conditions on the model primitives so that the space derivatives ∂ i v 0 ( t, x ) and ∂ i ∂ j v 0 ( t, x ) admit probabilistic representations as expectations of random processes { X x 0 , ∇ X x 0 , H [ X x 0 ] } that can be easily simulated.

3.2 Probabilistic Representation of the Space Derivatives

We proceed with introducing assumptions that guarantee Theorem 2, providing representations of ∂ i v 0 ( t, x ) and ∂ i ∂ j v 0 ( t, x ) as illustrated in (2.6). To achieve this, we first need to ensure that the derivative of the mapping x → X x 0 is well defined. This is formally established in Proposition A.1.

Assumption 3. For each z ∈ R d ′ 0 , the SDE coefficients ( s, x ) → ( µ 0 ( s, x ) , σ 0 ( s, x ) , χ 0 ( s, x, z )) are C 0 , 2 ([0 , T ] , R d ) . For each i, j = 1 , . . . , d , the coefficients and derivatives, seen as functions ( s, x ) → ( α ( s, x ) , β ( s, x ) , ζ ( s, x, · )) where ( α, β, ζ ) = ( µ 0 , σ 0 , χ 0 /γ ) , ( ∂ i µ 0 , ∂ i σ 0 , ∂ i χ 0 /γ ) , and ( ∂ j ∂ i µ 0 , ∂ j ∂ i σ 0 , ∂ j ∂ i χ 0 /γ ) are uniformly Lipschitz; i.e. there exists 0 ≤ ℓ < ∞ s.t. for all s ∈ [0 , T ] , z ∈ R d ′

Moreover, | ζ ( s, 0 , z ) | is uniformly bounded for s ∈ [0 , T ] and z ∈ R d ′ 0 .

In view of this assumption, we consider the following SDEs, as jump versions of (2.7), for which the strong solutions should be the space derivatives of X x 0 . Again, the dependence of the coefficients on r , X x 0 ( t, r -) , and z , as well as the dependence of X x 0 , ∂ a X x 0 , ∂ a ∂ b X x 0 on ( t, s ) , ( t, r -) has been

suppressed.

As we will show in Proposition A.1, under Assumption 3 the process X x 0 ( t, s ) has a version that is twice continuously differentiable in x for every 0 ≤ t < s ≤ T . The processes {∇ X x 0 , H [ X x 0 ] } , as defined in (3.5), will then correspond to the derivatives. Moreover, these processes, as well as X x 0 , will possess desirable integrability properties.

To guarantee sufficient integrability and to provide a variance bound for our estimator, we also need to assume growth conditions on the rewards.

Assumption 4. Assume that the mapping x → ρ 0 ( t, x ) , g 0 ( x ) is C 2 for all t ∈ [0 , T ] . Moreover, for h ( t, x ) = ρ 0 ( t, x ) and g 0 ( x ) there exists c h s.t.

With these assumptions, we validate the representations in (2.6) using the following theorem.

Theorem 2 (Probabilistic Representation of the Space Derivatives) . Under Assumptions 3 and 4, the representations in (2.6) hold with the jump version of { X x 0 , ∇ X x 0 , H [ X x 0 ] } in (3.1) and (3.5) .

3.3 The Generator Gradient Estimator

With Theorems 1 and 2, we construct our generator gradient estimator and show that it is unbiased with a variance that grows polynomially in x . Recall the estimators Z ( t, x ) and H ( t, x ) in (2.6).

By Theorem 2 and the integrability in Proposition A.1 under Assumption 3, the equality (2.8) holds. Then, following the notation in (2.9), we define

where ∇ θ L C 0 V 0 ( t, x ) and ∇ θ L J 0 V 0 ( t, x ) are defined by replacing ∂ i v ( t, x ) with Z i ( t, x ) and ∂ j ∂ i v with H i,j ( t, x ) in (3.3) and (3.4), respectively. Then, our estimator D ( x ) is given by (2.9).

Theorem 3. Suppose Assumptions 1-4 are in force. Then, the generator gradient estimator D ( x ) is unbiased; i.e. ED ( x ) = ∇ θ v 0 (0 , x ) . Moreover, the variance Var( D ( x )) ≤ C ( | x | +1) 2 m +4 has at most polynomial growth in x , where the constant C can be dependent on other parameters of the problem but not x .

Remark. The m signifies the growth rate of the rewards and their derivatives. The extra additive factor 2 in the variance is from the growth of the θ derivative of a 0 , the volatility squared.

4 Example: Linear System with Quadratic Loss

In this section, we illustrate some analytical properties and the effectiveness of our estimator by considering a linear quadratic control problem.

25

GeneratorGradient

5000

GeneratorGradient

PathwiseDifferentiation

95%CI

20

value

4000

PathwiseDifferentiation

ime

95%CI

i

run

15

stimatec

3000

2000

6

10

1000

S

5

0

0

-1000-

102

103

104

105

106

107

108

0

20

40

60

80

100

Dimension of θ

Coordinateindex ofθ

(a) Average runtime for different n .

(b) Estimator statistics with n = 102 .

Let X ∈ R d be the controlled process, given by the solution to the SDE

where B ( t ) ∈ R d ′ is a standard Brownian motion, U ( t ) ∈ R m is the control process that is adapted to the filtration generated by X , A ∈ R d × d , B ∈ R d × m , C ∈ R d × d ′ are non-random matrices. The objective is to choose an admissible control U ( t ) that minimizes the quadratic loss

where Q,Q T ∈ R d × d and R ∈ R m × m are non-random matrices.

In various applications of interests, the admissible control U ( t ) is a parameterized function of time and state U ( t ) = u θ ( t, X x θ ( t )) where the state process under control u θ is denoted by X x θ . The dimension of θ could potentially be very high-e.g. when u θ is a neural network. To achieve an optimized loss in this over-parameterized setting, one common approach is to run gradient descent. Hence, an efficient gradient estimator that scales well with the dimension n of θ is highly desirable.

We compare the performance of the proposed generator gradient estimator and the pathwise differentiation estimator. In this context, these estimators take the following form. The detailed derivations are presented in Appendix F.1.

The Generator Gradient Estimator: In this setting, our generator gradient estimator in (2.9) is

where the definition of Z follows from (2.6), and is given by (F.1) in Appendix F.1. As explained in (2.9), we also randomize the integral corresponding to the gradient of the reward rate ∇ θ ρ 0 .

The Pathwise Differentiation Estimator: From (1.3), we find the following IPA estimator that randomizes the time integral

Here, the pathwise derivatives ∂ θ i X x θ ( t ) is the solution to (F.3).

We deploy these estimators in an environment where the state variable x ∈ R 4 represents the x-y positions and velocities of a point mass on a 2D plane. The controller applies a force to this mass. The cost function is designed to encourage the controller to swiftly move the point mass to the origin with minimal force. The force is state-time-dependent and parameterized through a 4-layer fully connected neural network with variable width. All computation times are recorded from a Tesla V100 GPU. Further details about the setup of our numerical experiments can be found in Appendix F.2.

In Figure 1a, we present a comparison of the average runtime for computing a single sample of the generator gradient and the pathwise differentiation estimators D ( x ) , ˜ D ( x ) ∈ R n , across increasing values of n the dimension of θ . Our findings indicate that the generator gradient estimator not only outperforms the widely used pathwise differentiation method across all tested values of n but also surpasses it by more than an order of magnitude for larger values of n . Additionally, the computation time for our estimator shows remarkable stability with respect to increases in n , displaying only a slight uptrend when n ≳ 10 7 .

Figure 1b confirms that, at n = 102 , the estimated values by the two estimators are very similar with high confidence. This confirms that our estimator is consistently estimating the gradient ∇ θ v θ (0 , x ) .

Finally, Table 2 presents the standard errors (SE) (F.4) from 400 replications of both estimators, averaged over the gradient coordinates. It also displayed the averaged ratios of the standard errors (F.5). We observe averaged SE ratios that are consistently less than 1 for all n , suggesting that our generator gradient estimator not only provides significantly faster computations as shown in Figure 1a but also achieves lower estimation variances. Further analysis of the SEs for each gradient coordinate is conducted and displayed in Figure 2 in Appendix F.2, highlighting similar histogram shapes and observable reduction in large values of SEs of our estimator.

5 Concluding Remarks

The theoretical results in this paper have the limitation of requiring second-order continuous differentiability and uniform boundedness of the space derivatives of the parameters of the underlying jump diffusion. These strong conditions, which are standard in the literature of stochastic flows (cf. [19, 11]) to guarantee global existence and uniqueness of the derivative processes in (3.5), are necessary to achieve the generality of the results presented in this paper.

However, our generator gradient estimator often works even when coefficients are not continuously differentiable. This is true if the generator and rewards gradients are defined almost everywhere, and the derivative processes in (3.5), with almost everywhere derivatives of the SDE parameters, exist for every t ∈ [0 , T ] and satisfy some integrability conditions. Examples include neural networks parameterized stochastic control with ReLU activation functions, heavy-traffic limits of controlled multi-server queues, and the Cox-Ingersoll-Ross (CIR) model. For these models, the existence and integrability of the derivative processes can be checked on a case-by-case basis, allowing the consistency and unbiasedness of the generator gradient estimator to be established. We confirm this by numerically investigating the CIR process and an SDE with ReLU drift in Appendix G.

Acknowledgments and Disclosure of Funding

The material in this paper is based upon work supported by the Air Force Office of Scientific Research under award number FA9550-20-1-0397. Additional support is gratefully acknowledged from NSF 2118199, 2229012, 2312204, and ONR 13983111.

References

R. T. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud. Neural ordinary differential equations. Advances in Neural Information Processing Systems , 31, 2018.

L. C. Evans. Partial Differential Equations , volume 19. American Mathematical Society, 2022.

W. Fang and M. B. Giles. Importance sampling for pathwise sensitivity of stochastic chaotic systems. SIAM/ASA Journal on Uncertainty Quantification , 9(3):1217-1241, 2021.

P. W. Glynn. Optimization of stochastic systems via simulation. In Proceedings of the 21st Winter Simulation Conference , pages 90-105, 1989.

P. W. Glynn. Likelihood ratio gradient estimation for stochastic systems. Communications of the ACM , 33(10):75-84, 1990.

X. Guo, A. Hu, and Y. Zhang. Reinforcement learning for linear-convex models with jumps via stability analysis of feedback controls. SIAM Journal on Control and Optimization , 61(2): 755-787, 2023.

J. Jia and A. R. Benson. Neural jump stochastic differential equations. Advances in Neural Information Processing Systems , 32, 2019.

Y. Jia and X. Y. Zhou. Policy gradient and actor-critic learning in continuous time and space: Theory and algorithms. Journal of Machine Learning Research , 23(275):1-50, 2022.

I. Karatzas, S. E. Shreve, I. Karatzas, and S. E. Shreve. Methods of Mathematical Finance , volume 39. Springer, 1998.

P. Kidger. On neural differential equations. arXiv preprint arXiv:2202.02435 , 2022.

H. Kunita. Stochastic Flows and Jump-Diffusions . Springer, 2019.

X. Li, T.-K. L. Wong, R. T. Q. Chen, and D. Duvenaud. Scalable gradients for stochastic differential equations. In S. Chiappa and R. Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , volume 108 of Proceedings of Machine Learning Research , pages 3870-3882. PMLR, 26-28 Aug 2020.

T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y . Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 , 2015.

D. Lipshutz and K. Ramanan. A Monte Carlo method for estimating sensitivities of reflected diffusions in convex polyhedral domains. Stochastic Systems , 9(2):101-140, 2019.

D. B. Madan, P. P. Carr, and E. C. Chang. The variance gamma process and option pricing. Review of Finance , 2(1):79-105, 1998.

S. Massaroli, M. Poli, S. Peluchetti, J. Park, A. Yamashita, and H. Asama. Learning stochastic optimal policies via gradient descent. IEEE Control Systems Letters , 6:1094-1099, 2021.

R. C. Merton. Option pricing when underlying stock returns are discontinuous. Journal of Financial Economics , 3(1):125-144, 1976. ISSN 0304-405X.

B. Øksendal and A. Sulem. Applied Stochastic Control of Jump Diffusions . Springer, 2019.

P. Protter. Stochastic Integration and Differential Equations . Springer-Verlag, Berlin, Heidelberg, second edition, 1992.

J. Sirignano, J. MacArt, and K. Spiliopoulos. Pde-constrained models with neural network terms: Optimization and global convergence. Journal of Computational Physics , 481:112016, 2023.

R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems , 12, 1999.

F. Tröltzsch. Optimal Control of Partial Differential Equations: Theory, Methods, and Applications , volume 112. American Mathematical Soc., 2010.

B. Tzen and M. Raginsky. Neural stochastic differential equations: deep latent Gaussian models in the diffusion limit. arXiv preprint arXiv:1905.09883 , 2019.

Z. Wang and J. Sirignano. A forward propagation algorithm for online optimization of nonlinear stochastic differential equations. arXiv preprint arXiv:2207.04496 , 2022.

R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning , 8:229-256, 1992.

J. Yang and H. J. Kushner. A Monte Carlo method for sensitivity analysis and parametric optimization of nonlinear stochastic systems. SIAM Journal on Control and Optimization , 29 (5):1216-1249, 1991.

Appendices

A Generalizations of the Assumptions

A.1 Probabilistic Representation of the Gradient

In this section, we develop a generalized version of Theorem 1, weakening Assumptions 1 and 2. In particular, we allow discontinuities in time of the SDE coefficients. This flexibility is especially relevant in applications in data-driven decision-making environments where non-homogeneous SDE models with estimated drift, volatility, and jump parameters could be piece-wise constant. Moreover, we also relax the differentiability of the coefficients in the space variable to Lipschitz continuity. We will state the new set of assumptions, and establish a generalized version of Theorem 1 as in Theorem 1'

We proceed by presenting a critical theorem, along with the necessary assumptions, that forms the foundation of our probabilistic representation in Theorem 1'.

Assumption 5. Assume that for each θ the coefficients µ θ ( · , · ) , σ θ ( · , · ) , and χ θ ( · , · , · ) are jointly Borel measurable. Moreover, assume the following holds true:

At x = 0 , the coefficients are bounded: for all p ≥ 2

The coefficients are uniform Lipschitz in x , uniformly in s, θ in the following sense: there exists constants c and { c p : p ≥ 2 } s.t.

|

θ

µ

(

s, x

)

-

and for any p ≥ 2

for all s ∈ [0 , T ] , x, x ′ ∈ R d , θ ∈ Θ .

The coefficients are weakly Lipschitz in θ the following sense: for each p ≥ 2 , there exists a time-dependent positive field { κ p θ,θ ′ ( s ) ∈ R > 0 : s ∈ [0 , T ] , θ, θ ′ ∈ Θ } s.t. for some constant ℓ p ,

for all θ, θ ′ ∈ Θ , and the coefficients satisfy

|

p

θ

µ

(

and

p

θ,θ

′

θ

θ

θ

′

′

s, x

)

µ

(

s, x

)

κ

(

s

)(

x

+1)

,

σ

(

s, x

)

σ

(

s, x

-

|

|

|

|

-

)

|

κ

p

θ,θ

for all s ∈ [0 , T ] , θ, θ ′ ∈ Θ , x ∈ R d .

̸

Theorem K (Theorem 3.3.1 of Kunita [11]) . If Assumption 5 is in force, then the family of solutions X x := { X x θ ( t ) : t ∈ [0 , T ] , θ ∈ Θ } has a version ˆ X x (i.e. { ∃ t ∈ [0 , T ] , θ ∈ Θ : X x θ ( t ) = ˆ X x θ ( t ) } ⊂ N with P ( N ) = 0 ) that is B ([0 , T ]) ×B (Θ) ×F measurable. Moreover, w.p.1, for each θ ∈ Θ , X x θ ( ω, t ) is cadlag in t , and θ → ˆ X x θ ( ω, · ) seen as a mapping Θ → ( D [0 , T ] , ∥ · ∥ ∞ ) is uniformly continuous on compacts. Furthermore, for any p ≥ 2 there exists b p ∈ (0 , ∞ ) s.t.

p

µ

θ

(

′

s, x

)

c

|

x

-

x

′

|

,

|

σ

θ

(

s, x

)

-

σ

θ

(

′

s, x

)

c

|

x

-

x

′

|

,

p

′

(

s

)(

|

x

|

+1)

p

,

Remark. Theorem K is an extension of Kunita [11, Theorem 3.3.1] using the a.s. version of Kolmogorov's continuity criterion; see Corollary 1 of Protter [19, Theorem 73].

To guarantee that Theorem 1' holds, the requirement that Assumption 5 holds for all p ≥ 2 can be relaxed to all p ≤ n + ϵ where n is the dimension of θ . However, the intended application of our theory focuses on a regime where n ≫ d . So, we adopted this stronger version of Assumption 5. This also clarifies the presentations of the following assumptions: To guarantee the main results of this paper, a weaker version of this assumption requires that the assumption holds for all p ≤ 4 m + ϵ , where m is the growth rate of v, r, g , and their derivatives as in Assumption 8 and 7.

Next, we present additional regularities that implies the probabilistic representation in Theorem 1'.

Assumption 6. For each s ∈ [0 , T ] and z ∈ R d ′ 0 , the mappings ( θ, x ) → µ θ ( s, x ) , σ θ ( s, x ) , χ θ ( s, x, z ) , ρ θ ( s, x ) , g θ ( s, x ) are C 1 , 0 (Θ , R d ) .

Assumption 7. The measurable reward rate ρ θ and terminal reward g θ functions are Lipschitz in θ in the following sense:

There exist m ≥ 1 , α > 1 , and { κ α θ,θ ′ ( s ) ∈ R > 0 : s ∈ [0 , T ] , θ, θ ′ ∈ Θ } s.t.

for all θ, θ ′ ∈ Θ , and the reward rate satisfies

for all s ∈ [0 , T ] , θ, θ ′ ∈ Θ , x ∈ R d .

For some ℓ ≥ 0 , the terminal reward satisfies

Note that for notation simplicity, w.l.o.g. we use the same κ α θ,θ ′ and ℓ α as in part 3 of Assumption 5 to denote the Lipschitz coefficient, and the same m as in Assumption 9.

Remark. It is not hard to see that Assumptions 6 along with 5 and 7 are generalization of Assumption 1; i.e. if Assumption 1 holds then so will Assumptions 6, 5, and 7.

Next, we slightly generalize the growth part in Assumption 2 as in Assumption 9.

Assumption 8. Assume that { v θ ∈ C 1 , 2 ([0 , T ] , R d ) : θ ∈ Θ } is a family of classical solution to the PIDEs

where L θ = L C θ + L J θ as defined in (3.2) .

Assumption 9. There exists 0 < c v < ∞ and m ≥ 1 s.t.

Moreover, for each θ , there exists c θ,v s.t.

Theorem 1'. If Assumptions 5, 6, 7, 8, and 9 are in force, then the statement in Theorem 1 hold; i.e.

where ∇ θ L 0 := ∇ θ L C 0 + ∇ θ L J 0 are define in (3.3) and (3.4) , respectively.

A.2 Probabilistic Representation of the Space Derivatives

Following the same spirit, in this section, we develop a generalized version of Theorem 2, weakening the Assumption 3 to the following Assumption:

Assumption 10. For each s ∈ [0 , T ] , z ∈ R d ′ 0 , the coefficients x → ( µ 0 ( s, x ) , σ 0 ( s, x ) , χ 0 ( s, x, z )) is second continuously differentiable. Moreover, for each i, j = 1 , . . . , d , the coefficients and derivatives, seen as functions ( s, x ) → ( α ( s, x ) , β ( s, x ) , ζ ( s, x, · )) where ( α, β, ζ ) = ( µ 0 , σ 0 , χ 0 ) , ( ∂ i µ 0 , ∂ i σ 0 , ∂ i χ 0 ) , and ( ∂ j ∂ i µ 0 , ∂ j ∂ i σ 0 , ∂ j ∂ i χ 0 ) satisfies the following conditions:

At x = 0 , the coefficients are uniformly bounded in time: for all p ≥ 2

and for any p ≥ 2

for all s ∈ [0 , T ] , x, x ′ ∈ R d

Proposition A.1. Suppose that Assumption 10 is in force. Then, the family of stochastic flow solutions { X x 0 ( s, t ) , 0 ≤ s ≤ t ≤ T : x ∈ R d } of the SDEs (3.1) has a version ˆ X 0 that is second differentiable in x at any time. Moreover, { ˆ X x 0 , ∇ ˆ X x 0 , H [ ˆ X x 0 ] : x ∈ R d } is a version of the solutions of the systems of SDEs (3.1) and (3.5) . Further, the family of solutions of (3.1) and (3.5) satisfies the following properties:

For each p ≥ 1 , there is 0 < b p < ∞ s.t.

and the derivatives

For any p ≥ 1 , there exists 0 < l p < ∞ s.t. for all x, x ′ ∈ R d ,

A proof of Proposition A.1 is provided in Appendix C.

Remark. We observe that part 2 of Assumption 10 will imply the space derivatives of the coefficients are bounded, which is used to get the L p boundedness and Lipschitzness of the derivative processes. Assumption 10 also ensures that the second derivative is uniformly Lipschitz as well. This is not used in the proof for the upcoming results.

We establish the following Theorem 2' generalizing 2. The proof is deferred to Appendix D.

Theorem 2'. Under Assumptions 10 and 4, then (2.6) holds; i.e.

where { X x 0 , ∇ X x 0 , H [ X x 0 ] } are the strong solutions to (3.1) and (3.5) .

s.t.

A.3 The Estimator

With Proposition A.1 and Theorem 2', we are ready to define our generator gradient estimator for ∇ θ v 0 (0 , x ) and show that it is unbiased and has a variance that grows polynomially in x .

First, recall the definition of Z and H in (2.6)

Observe that by Theorem 2', the integrability in Proposition A.1,

a similar property hold for the H ( t, x ) process as well. Therefore, we can replace the derivatives ∇ v 0 with Z and H [ v 0 ] with H in Theorem 1' without changing the expectation, showing the validity of (2.8). In particular, this implies that the generator gradient estimator defined in (2.9) is unbiased.

Finally, we establish a generalized version Theorem 3 using the assumptions in this section. The additional proof of this theorem is presented in Appendix E.

Theorem 3'. Suppose Assumptions 4-10 are in force. Then, the generator gradient estimator (2.9) is unbiased; i.e. ED ( x ) = ∇ θ v 0 (0 , x ) . Moreover, if the α > 1 in item 1 of Assumption 7 is replaced by α > 2 , then the variance Var( D ( x )) ≤ C ( | x | +1) 2 m +4 has at most polynomial growth in x , where C can be dependent on other parameters of the problem but not x .

Remark. The m signifies the growth rate of the rewards and their derivatives. The extra additive factor 2 in the variance is from the growth of the θ derivative of a 0 , the volatility squared.

B Proof of Theorem 1'

In this section, we prove Theorem 1' and hence the simplified Theorem 1.

where the operators L C θ and L J θ are defined in (3.2).

Then, an application of Itô's formula (B.1) under Assumption 8 and 9 yields the following result for which the proof is presented in Section B.1.

Lemma 1. For any θ ∈ Θ ,

are martingales for 0 ≤ t ≤ T .

Therefore,

Then, rearranging terms, one gets

where ( i ) follows from Assumption 8 that v θ ( T, · ) = g θ ( · ) and ρ 0 ( s, x ) = -∂ s v 0 ( s, x ) -L 0 v 0 ( s, x ) for all x ∈ R d and 0 ≤ s < T , and ( ii ) recalls the definition that L θ = L C θ + L J θ .

To conclude Theorem 1', we analyze the finite difference approximations of the above three expectations, where they correspond to the continuous , the jump , and the rewards part, respectively. The results are summarized by the following Proposition B.1, whose proof is deferred to Appendix B.2.

Proposition B.1. Under the assumptions of Theorem 1', for K = C, J ,

where ∇ θ L C 0 and ∇ θ L J 0 are defined in (3.3) and (3.4) respectively. Moreover,

and

With Proposition B.1 handling each term in (B.2), we conclude that

B.1 Proof of Lemma 1

Apply Itô's formula (B.1) to v θ ( t, X x θ ( t )) yield

by Assumption 8, where I 1 ( t ) and I 2 ( t ) denotes the Itô stochastic integrals on the previous lines, respectively. Since the integrands are a.s. finite, I 1 ( t ) and I 2 ( t ) are local martingales. We show that they are true martingales. First, for I 1 , apply the Burkholder-Davis-Gundy inequality

where C is some constant that could change line by line. Notice that by Assumption 5,

Therefore, by Assumption 5 item 2, Assumption 9, and Theorem K

Therefore, I is a martingale. For I , by Kunita [11, Proposition 2.6.1]

where ( i ) follows from the mean value theorem with ξ := ξ θ ( s, X x θ ( s ) , z ) ∈ [0 , 1] , and ( ii ) follows from the Cauchy-Schwartz inequality applied to the integral w.r.t. the finite measure P × Leb × µ . By Assumption 5,

Again, by Assumption 5 item 2 and Theorem K,

Also, by Assumption 9 and ξ ∈ [0 , 1]

where ( i ) follows from µ being a finite measure and γ ( z ) = | z | ∧ 1 ≤ 1 . This shows that I 2 , hence M θ,θ ( t ) is a martingale.

To show that M 0 ,θ ( t ) is a martingale, we employ the same derivation with v θ replaced by v 0 . This completes the proof of Lemma 1.

B.2 Proof of Proposition B.1

Since X x is indistinguishable from ˆ X x , we can use X x and ˆ X x interchangeably when evaluating expectations. Therefore, it is understood that we use ˆ X x when we need continuity in θ , while we keep the notation X x .

In this proof, for notation simplicity, the letter C will denote a constant that could change from line to line. C can be dependent on the dimensions d, d ′ , the growth rate m , the horizon T , the Lévy measure ν , and polynomial power p or α . But it doesn't depend on θ (or sometimes δ ) and x .

The Continuous Part: We prove the claim that the derivative for the continuous part should be

where ∇ θ L 0 is defined in (3.3).

To proceed, we also claim that

so that the derivative ratio and the derivative are well-defined. The finiteness of these expectations is shown below.

To prove the claimed expression (B.4) is indeed the derivative, we consider the limit

To show the r.h.s. go to 0, we first show what's inside the two integrals is U.I. Consider for α > 1 ,

For the first term, consider

where ( i ) follows from Assumption 9. For the second term, recall the definition in (2.1):

The two terms can be handled in the same way as follows:

where ( i ) follows from Assumption 5 item 2 and the constant bound for σ θ ( t, 0) in (B.3). Going back to inequality (B.7), these bounds implies that

for some C that doesn't depend on θ , where ( i ) follows from item 3 of Assumption 5. Therefore,

where ( i ) applies Fubini's theorem due to the positivity of κ α θ, 0 , and ( ii ) follows from Theorem K. We have shown that this expectation above is finite and independent of θ . Note that, in particular, this implies that the first expectation in (B.5) is finite as well.

For the second term in the last line of (B.7), we first consider for matrix M ∈ R n × d and vector v ∈ R d ,

Apply this inequality, we obtain

where ( i ) uses Assumption 9 and the previous matrix norm inequality, and ( ii ) uses Cauchy-Schwartz inequality. Let e l ∈ R n be the unit vector with the l 'th coordinate equal to 1. Now by Assumption 5,

we have that for fixed ϵ > 0 and l = 1 , 2 , . . . , n

By the same argument as in (B.9), this is uniformly bounded in δ . Hence

where, again, by the same argument as in (B.9), ψ 1 is chosen to be finite. For the second term in the last line of (B.10), we consider the quantity

where ( i ) follows from Assumption 5 and ( ii ) applies Jensen's inequality and ( iii ) recalls the definition in (B.3). Note that the reason we define ϕ ( p, δ, t, x ) is because

From (B.12), we see that the same argument in (B.9) implies the [0 , T ] × Ω integrability of ϕ (2 α + ϵ, δ, s, X x 0 ( s )) , uniformly in δ . This then implies that ϕ (2 α, · , s, X x 0 ( s )) is U.I. for δ in a neighbourhood of 0 . Therefore, we see that

Combining these with (B.10), we have establish that

In particular, this shows the second expectation in (B.5) is finite as well.

Therefore, in view of (B.7), (B.9), and (B.14), we conclude that

is U.I. on (Ω × [0 , T ] , F × B ([0 , T ]) , P × 1 T Leb) . Hence,

We use the mean value theorem to get that for some C > 0 and ξ i = ξ θ,i ( s, X x θ ( s )) ∈ (0 , 1) , η i,j = η θ,i,j ( s, X x θ ( s )) ∈ (0 , 1) ,

where the last equality follows from the continuity of ( θ, x ) → ∇ θ µ θ,i ( s, x ) and ∇ θ a θ,i,j ( s, x ) , x → ∂ i v 0 ( s, x ) (Assumption 6) and ∂ i ∂ j v 0 ( s, x ) (Assumption 8), and θ → X x θ ( s ) (Theorem K).

Therefore, going back to the limit ratio in (B.6), we have shown that

The Jump Part: Similar to the continuous part, we claim that the derivative should be

where ∇ θ L J 0 is defined in (3.4).

To simplify notation, write

where

Further, we write χ θ := χ θ ( s, X x θ ( s ) , z ) and χ θ,i := χ θ,i ( s, X x θ ( s ) , z ) when there is no ambiguity in the dependence on s, X x θ ( s ) , z . Then, apply the mean value theorem to ρ → v 0 ( s, X x θ ( s ) + ρχ θ + (1 -ρ ) χ 0 ) , there exists ξ = ξ θ ( s, X x θ ( s ) , z ) ∈ (0 , 1) s.t.

Therefore,

Again, we consider the limit

where, as we will show below, the two pre-limit expectations in the first line are finite.

As before, we proceed show that the limit in θ can be exchanged into the triple integration by showing U.I. of

on Ω × [0 , T ] × R d 0 w.r.t. the probability measure P × 1 T Leb × 1 µ ( R d ′ 0 ) µ . We consider

(B.19)

We consider the two terms separately. For E 1 , applying the mean value theorem again to (B.16), there exists η i = η θ,i ( s, X x θ ( s ) , z, ξ ) s.t.

Therefore,

We look at I 1 and I 2 separately. By Assumption 9,

where we recall that γ ( z ) = | z | ∧ 1 ≤ 1 . Then, we consider, by Assumption 5, for p ≥ 2

and for all θ, θ ′ ∈ Θ

So, for p ≥ 2

As µ is a finite measure, we have

For I 2 , we bound

By Assumption 5,

Use this and inequality (B.21), we obtain

In summary, we have

Therefore, by the same argument as in the derivation (B.9), we conclude that

where ( i ) uses Jensen's inequality and Theorem K. Note that, with α = 1 , this also implies the finiteness of the first expectation in (B.17).

For the second term in (B.19), we use the same technique as in the derivation for that of the continuous part. First, we consider

where ( i ) applies the mean value theorem to ρ → ∂ i v 0 ( s, x + ρχ 0 ) to yield the existence of such ξ i := ξ i ( s, x, z ) , ( ii ) follows from Assumption 9, and ( iii ) uses Hölder's inequality ∥ fg ∥ 1 ≤ ∥ f ∥ ∞ ∥ g ∥ 1 as well as γ ( z ) ≤ 1 , and ( iv ) follows from (B.21) where we have that for p ≥ 2

Therefore, by Theorem K and Cauchy-Schwarz inequality,

To bound this, as in the continuous part, we check the uniform integrability on Ω × [0 , T ] × R d ′ 0 w.r.t. the probability measure P × 1 T Leb × 1 µ ( R d ′ 0 ) µ when δ is in a sufficiently small neighbourhood of 0 of the derivative ratio

To simplify notation, we again denote χ θ,i := χ θ,i ( s, X x 0 ( s ) , z ) . To check this, we consider for ϵ ≥ 0

independent of δ . Choose ϵ > 0 will show the U.I. of (B.23). Therefore, we

where the last inequality follows from previous derivation with ϵ = 0 . In particular, recalling the definition of E 2 in (B.19), this shows that

as claimed above.

Therefore, by bounding the two terms in (B.19), we conclude the uniform integrability of (B.18). So, going back to (B.17), U.I. implies that

where the last step follows from

Here, ( i ) applies the mean value theorem and ( ii ) use the continuity of θ →∇ θ χ ξ i θ,i ( s, X x θ ( s ) , z ) as in Assumption 6.

The Rewards Part: We first consider the reward rate r . As in the previous proof, we show the U.I. of

and the finiteness of

for some α > 1 . By Assumption 7 item 1 and the same derivation as in (B.9),

uniformly in θ . Moreover,

These results and the continuity of ( θ, x ) →∇ θ r ( s, x ) and θ → X x θ ( s ) implies that

For the terminal reward g term, the same proof with the integral removed and s replaced by T will yield the desired conclusion.

C Proof of Proposition A.1

We note that the statement for X x 0 and its first derivative holds from directly applying Kunita [11, Theorem 3.3.2] and the a.s. version of Kolmogorov's continuity criterion as in Corollary 1 of Protter [19, Theorem 73].

To show the statement for the second derivative, we apply the proof of Kunita [11, Theorem 3.4.2]. From display (3.43), we look at the random drift:

seen as a function of r ∈ [0 , T ] , H ∈ R d × d × d , and show that it satisfies the conditions for Kunita [11, Theorem 3.3.2] with λ = x ; i.e. the conditions in Assumption 10.

We note that as µ 0 and ∂ m µ 0 ( r, x ) satisfying item 2 of Assumption 10 for any l, m = 1 , . . . , d ,

is bounded. Same for ∂ m ∂ l µ 0 .

First, at H a,b = 0 .

Second, M x a,b,i ( r, H a,b ) is clearly uniformly Lipschitz in H a,b as ∂ l µ 0 is bounded.

Third, using the boundedness of ∂ m µ 0 and ∂ m ∂ l µ 0 , we have

and

Therefore, defining K x,y ( r, H ) to be the sum of the two, we see that

Here the first inequality follows from the first derivative satisfying the Proposition A.1, which follows from a direct application of Kunita [11, Theorem 3.3.2].

Similar results can be established for the volatility and the jump coefficients. Therefore, we conclude the proof by applying Kunita [11, Theorem 3.3.2] to the derivative and the second derivatives.

D Proof of Theorem 2'

Our proof of Theorem 2' hinges on the ability to exchange the derivative with the expectation and time integral. To achieve this, first, we use similar techniques as in the proof of Theorem 1' to prove the following lemma.

Lemma 2. Under the assumptions of Theorem 2', for h ( t, x ) = ρ 0 ( t, x ) and g 0 ( x ) , we have

and

Moreover, there exists a constant C independent of t, s s.t.

Lemma 2 directly implies that the derivatives of the expected terminal rewards in (2.6) satisfy

By the same argument, to prove Theorem 2', it suffices to show that for the cumulative reward parts in (2.6), the time integral and space derivatives can be interchanged. First, by Lemma 2, we see that

So, by Fubini's theorem and Lemma 2

where ( i ) follows from dominated convergence that for y in a ϵ neighbourhood of x ,

independent of s . Similarly,

This and (D.3) implies (2.6), completing the proof.

D.1 Proof of Lemma 2

First Space Derivatives: We first show equality (D.1). Consider

We exchange the limit and the expectation by considering

where the mean value theorem implies the existence of such r.v. ξ ∈ [0 , 1] . For the first term, Proposition A.1 implies that

For the second term, by Assumption 4

where the last inequality considers | δ | ≤ 1 and C can be chosen so that it doesn't depend on δ , s , and t . Therefore, the limit in the r.h.s. of (D.4) can be interchanged with the expectation and we have that

Also, the previous derivation with α = 1 and taking the limit δ → 0 implies that

where C doesn't depend on s and t .

Second Space Derivatives: Then, we show equality (D.2). Previous proof implies that

Hence we employ the same strategy to exchange the limit and expectations for the following expression

We show U.I. for the two terms in (D.6) separately. For the first term, consider

By Proposition A.1, the first expectation is bounded uniformly in s and t . For the second term, consider

where ( i ) follows from the mean value theorem with r.v. ξ i ∈ [0 , 1] , and ( ii ) applies Assumption 4. Therefore, we have that

where the last inequality follows from Proposition A.1. This is uniformly bounded in δ as δ → 0 , showing U.I. for the first term in (D.6).

For the second term in (D.6), we consider

which is also uniformly bounded in δ as δ → 0 .

Therefore, exchanging the limits in (D.6), we obtain

Moreover, by setting α = 1 and taking the limit as δ → 0 in the preceding derivations, we see that

where the constant C is uniform in s and t .

E Proof of Theorem 3'

From (A.1), we see that

Moreover, since τ is independent of F ,

Therefore, by Theorem 1', ED ( x ) = ∇ θ v 0 (0 , x ) .

For the variance, we consider

By (B.11) and (B.13) with α = 2 ,

.

and similarly

So, we have that

where ( i ) uses α > 0 so that the integrand is U.I. in θ ∈ Θ (see (B.11) for a similar proof), and ( ii ) uses Jensen's inequality with 2 /α < 1 . The same holds for the terminal reward term, with κ α θ,θ ′ = ℓ α | θ -θ ′ | α integrable.

Therefore, we conclude that Var( | D ( x ) | ) ≤ E | D ( x ) | 2 ≤ C ( | x | +1) 2 m +4 , where C can be dependent on other parameters but not x .

F Supplementary Materials for Section 4

F.1 Calculating the Estimators

The Generator Gradient Estimator: We compute

By definition and Proposition A.1, we have that

Similarly, E | H ( t, X x 0 (0 , t )) | 4 ≤ C ( | x | +1) 4 m . These calculations imply that

For the reward rate and terminal reward terms, we recall Assumption 10 with the additional Assumption that α > 2 . Note that since α > 2 , for

Here ∂ i X x θ ( t, s ) is a column vector. So, replacing it by the Jacobian will yield a row vector. Following the definition of Z in (2.6), we define

Here, the derivative process ∇ X x θ satisfies the following ODE with random coefficients:

or in matrix form:

Therefore, in this setting, our generator gradient estimator in (2.9) is

where Z is given by (F.1). As explained in (2.9), we also randomize the integral corresponding to the gradient of the reward rate ∇ θ ρ 0 .

The Pathwise Differentiation Estimator: From (1.3), we construct the following IPA estimator that randomizes the time integral

Here, the pathwise derivatives ∂ θ i X x θ ( t ) is the solution the following ODE with random coefficient:

F.2 Numerical Experimentation Details

We conducted the computation time and variance comparison for both estimators using PyTorch. The computation time data was generated on a system equipped with a PCIE version of Nvidia Tesla V100 GPU, featuring 32GB of VRAM. Additionally, the system includes a 2-core CPU and 16GB of RAM, which are used to format and store data. The primary computational tasks are handled by the GPU.

The data for Table 2 is produced as follows. For each n , we produce 400 i.i.d. GG and PD estimators { D ( j ) ( x 0 ) , ˜ D ( j ) ( x 0 ) ∈ R n : j = 1 , . . . , 400 } . Let

The 'Avg SE of GG" and 'Avg SE of PD" entries record

respectively. The 'Avg SE ratios" compute

n = 1.02e+02

n = 1.00e+03

10-1

Pathwise Differentiation

Generator Gradient

10-3

10-4

n=5.50e+03

n = 2.10e+04

10-1

10-2

Density

10-3

10-4

10-5

n = 3.24e+05

n = 1.29e+06

10-2

Density

10-4.

10-6

n = 5.14e+06

n = 1.15e+07

10-2

Density

10-4.

10-6

0

100

200

300

0

100

200

300

Standard Error

Standard Error

The numerical values used for the matrices, initial conditions, and network initializations for the SDE models can be found in the supplied code.

We further analyze variance by plotting histograms of the distribution formed by the standard errors of the coordinates of the estimators, as shown in Figure 2. The standard error distribution of the pathwise differentiation method exhibits a heavier tail compared to our proposed generator gradient estimator. This aligns with the superior variance performance of our estimator demonstrated in Table 2. Figure 2 also provides insights into the confidence intervals in Figure 1b, which are barely visible due to high confidence levels. In particular, the generator gradient estimator has tighter confidence intervals in Figure 1b.

G Experiments on SDEs with Non-Differentiable Parameters

G.1 CIR Model

In this section, we use the Cox-Ingersoll-Ross (CIR) diffusion as an example to test the validity of the proposed generator gradient estimator when the differentiability assumptions of the coefficients are violated. Specifically, consider the one-dimensional process:

for t, s ∈ [0 , 2] , where x, θ > 0 . Note that the volatility σ ( t, x ) = √ x is not differentiable at 0, though it is C ∞ for x > 0 . A unique strong solution to (G.1) always exists. Moreover, if θ ≥ 1 / 2 , then X θ ( t ) > 0 for all t ∈ [0 , 2] almost surely.

We consider the following value function:

We aim to estimate the gradient ∇ θ v θ (0 , x ) evaluated at x = 0 . 1 for multiple values of θ .

Since the pathwise differentiation estimator also suffers from non-differentiability issues, we validate the generator gradient (GG) estimator by comparing it with the finite difference (FD) estimator. Specifically, the FD estimator computes

using Monte Carlo simulation of the SDE (G.1) and the value function. In this context, the FD estimator should consistently estimate the gradient evaluated at any θ > 0 . The GG estimator is produced from (2.9). We use the Euler scheme to simulate the SDEs and the derivative processes. To avoid numerical issues when the Euler discretization of the CIR process crosses 0, we take the absolute value of the discretized process at each time step.

Table 3 summarizes the estimated value and confidence interval for both the GG and FD estimators. We note that a bias of O ( h 2 ) is present in the FD case. When θ ≥ 1 / 2 , we observe that even though Assumptions 1 and 3 are violated, the GG estimator produces results consistent with the FD estimator. This suggests the validity of the GG estimator even when Assumptions 1 and 3 don't hold. This consistency occurs because, in this case, the derivative processes are still well-defined up to the first time X x θ ( t ) hits 0, which does not happen when θ > 1 / 2 .

However, when θ < 1 / 2 (cases highlighted in blue in Table 3), the sample paths of the SDE (G.1) can reach 0. Although the statistics in Table 3 appear consistent, we observe a significant increase in the variance of the GG estimator as θ decreases. This increase may indicate that the GG estimator is not consistently estimating the gradient in these cases.

G.2 SDE with ReLU Drift

In this section, we use the following SDE with ReLU drift as an example to test the validity of the proposed generator gradient estimator:

for t, s ∈ [0 , 2] , where θ > 0 and we choose x = -0 . 1 . Note that the +1 in the drift makes it always positive. So, starting from -0 . 1 , the process should cross 0 (where the drift is non-differentiable) before time 2 with high probability.

With v θ defined in (G.2), we aim to estimate the gradient ∇ θ v θ (0 , x ) evaluated at x = 0 . 1 for multiple values of θ using the GG and FD (defined in (G.3)) estimators.

Table 4 summarizes the estimated values and confidence intervals for both the GG and FD estimators. Note that a bias of O ( h 2 ) is present in the FD case. Despite violations of Assumptions 1 and 3, the GG estimator still produces consistent results compared to the FD estimator. We note that in this context, it should be possible to establish the existence and integrability of the derivative processes for any value of θ .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The context and contributions of this paper are clearly and accurately stated in the abstract and the introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We acknowledge the specific parts of the assumptions that could potentially be stronger than what is necessary. See, for example, the remarks and discussion following Assumption 1 and 2.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All theorems, propositions, and lemmas are either proved within the paper or cited from other works. The assumptions are clearly labeled and discussed. We also provided intuitive justification for our theoretical results in Section 2.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The code attached to the submission, if run properly with system configurations similar to that indicated in the appendix, will produce very similar qualitative results as that presented in the paper. However, as the neural networks are randomly initialized, the quantitative outcome might differ.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in

some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is submitted with the paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer:[Yes]

Justification: The parameters of the control system in Section 4 are presented in the code. There is no hyperparameter that needs fine-tuning.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: Although we do report data involving error bars (see 1b), these are barely visible, and have no qualitative significance to the numerical results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: This is fully specified in Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We conform to the NeurIPS Code of Ethics. Anonymity is preserved in this submission.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This work makes methodological contributions to the SDE gradient estimation problem. There is no direct social impact associated with this work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer:[NA]

Justification: There is no high-risk data in this paper.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA] .

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe article studies classical stochastic control in the continuous d-dimensional SDE setting (with and without jumps). The authors suggest to compute the gradients of the value function (expected total reward in the non-discounted finite time regime), which can be applied (in a model-based setting) to policy gradient type algorithms popular in RL. While in a straight-forward fashion computing the derivatives of the expectations would lead to the simulation of n (dimension of approximation space for the control) expectations that require the simulation of one SDE each, the authors propose a clever interchange of derivatives that reduces the number of SDEs to a polynomial in d. Very careful proofs are given, the findings are applied to simple linear SDE example with quadratic cost function.\n\nSTRENGTHS:\nThe article introduces a creative switching trick (switching order of derivatives and derivatives with expectations) that allows to reduce the simulation dimension compared to a straight-forward estimator.\n\nThe rest of the paper consists of a rigorous verification of interchanging limits (derivatives) and expectations. The proofs are careful (mainly based on differentiability properties of SDE flows), I spotted only few typos.\n\nWEAKNESSES:\nI do not agree with the comparison to RL (=sample based stochastic control) in Section 1.1. The entire point of RL is to provide model-free estimators, in particular, the policy gradient theorem cancels out the model transitions. Here, the situation is different. The model ($\\mu$ and $\\sigma$) is used to write down the SDEs (2.7) required to simulate the estimator. This is not a problem but shifts the article more towards the stochastic control community.\n\nI cannot see a clear interest in the ML/AI community in the questions raised, the control setting of SDEs (jump SDEs) is natural in the probability/math finance community but not at all in ML/AI. Even if I try I struggle to come up with examples relevant for the ML/AI community (the authors neither).\n\nThe article is written for a purely mathematical audience with research level skills in stochastic process theory, the article could appear almost unchanged in journals such as SPA or SIAM Journal on Control and Optimzation. I think NeurIPS is not the right place for publication. I am willing to increase my scores if a more relevant example than the quadratic problem could be provided that shows value of the switching trick to the ML/AI community.\n\nThe appendix is chaotic. It is tricky to follow the proofs because parts of the proofs are deferred to other sections of the appendix. Given that the proof only justifies interchange of limits and expectations it looks much harder than it is.\n\nQUESTIONS:\n1. You will typically have to solve the three SDES using numerical schemes. Can you provide additional assumptions on the model (the coefficients) that, combined with SDE numeric results (let's say in the Brownian case only), provides a result for the estimation error of the gradient? \n\n2. The additional derivatives on the model coefficients make me doubt the SDE scheme (2.7) works well if the coefficients are not so nice. Many control problems in applications are Heston type models, which are already problematic for numerics with the square-root variance coefficient. I know they do not satisfy the assumptions imposed, did you still try your simulation scheme?\n\n3. I do not understand why you included the jumps without an application in mind. This is normal for a maths journal, but a bit strange in ML/AI. Could you please provide an example?\n\n4. A few minor points:\n(i) The generator of a Markov process is the action plus the domain. Could you please be a bit more careful when you rework the article?\n(ii) You might want to mention that uniformly bounded derivatives imply linear growth which is the standard property required for global existence.\n(iii) I am probably missing where you discuss properties on the model that imply Assumption 2. If not I find assumptions on the derivatives of the mean rewards disturbing that I do not know how to check.\n(iv) Typo in Assumption 1, 2.\n(v) At least for the larger expectations it would help to be more consistent with the use of brackets.\n(vi) Could you please put references to the quadratic loss examples? Since this is the only application it would be good to see where and how it is used.\n(vii) the use of colons is not very helpful, for instance in (2.6) it is a bit complicated to understand what object is defined (I guess you want to define Z).\n(viii) The chain rule is missing in (D.3)\n\nLIMITATIONS:\nThe authors did not discuss the limitations or the practical problems of the results.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-14"}

comments: SUMMARY:
The paper proposes a novel algorithm for the gradient of machine learning objectives that are based on SDE paths, w.r.t. parameters. The method is scalable for big parameters, as the computational complexity of the estimation is not related to the number of parameters. Empirical results show the scalability advantage over classical pathwise differentiation.

STRENGTHS:
PDE constrained optimization has remained a very difficult question due to the high cost of simulating PDEs, let alone taking derivatives. Therefore, despite its many good theoretical promises, it has not been deployed. This paper attacks the question by reducing the computational cost with a monte-carlo estimation of the derivative, largely advances the promise of PDE constrained optimization. 
The writing is also very good, and the intuitions behind the methods are well illustrated.

WEAKNESSES:
1. Despite the fact that the computational complexity is irrelevant to the number of parameters, I suspect that as the number grows, the method would require more samples for an accurate monte-carlo estimate of the gradients.
Is it possible to either:
1. Provide a theoretical analysis on the convergence rate of the estimation w.r.t. the number of parameters.
2. Provide an empirical analysis on the convergence rate of the estimation w.r.t. the number of parameters.

QUESTIONS:
See weakness. Also, applying the Feynman-Kac twice requires the objective to be very smooth, can you provide some insights on the applications where this holds true?

LIMITATIONS:
As written in questions, the smoothness requirements induced by Feynman Kac etc. will limit the use cases.
As written in weaknesses, it is still unclear the convergence w.r.t. high number of parameters.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper formulates an efficient, unbiarsed, and finite variance gradient estimator for an objective function that looks like the stochastic optimal control cost function (it is ubiquitous across various applications). The problem concerns overparametrized SDEs (the paramete dimension n r is of a much higher dimension than the state space d).  The algorithm runs at a complexity invariant to the large n. The authors use the Feynman-Kac PDE for v_{\\theta} and transform it to an equivalent PDE to solve. They then employ a pathwise differentiation estimator to estimate the 2nd order and 1st order derivatives in the PDE while exploiting certain properties of the Hessian.\n\nSTRENGTHS:\n1. The applications of gradient estimators and the motivation behind this problem is well stated and supported. \n2. The methodology is clear, especially when finding probablistic representations of the gradient estimator.\n\nWEAKNESSES:\n1. The authors discuss the limitations in the checklist but it would have been more clear to have a limitations section clearly outlined as claimed in the checklist.\n2. The assumptions may be restrictive as mentioned. \n3. The work could be improved if there was a numerical simulation. The example of LQR is an illustrative example and it may be of didactic use and potentially relatively easy to implement a scenario when the parametrization is of a much higher dimension. \n4. The spatial derivatives \\partial v_0 are still difficult to compute, but still better than \\partial v_{\\theta}. What are some error bounds you can get by using the later derived probabilistic representations that can use Monte Carlo\n\nQUESTIONS:\n1. Are there any error bounds from using the probabilistic representations for the spatial derivatives (see #4 in Weaknesses)\n\nHappy to revise score if necessary.\n\nLIMITATIONS:\nYes they have mentioned some, rest may be in the above Weakness section unless there are some misunderstandings.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}