An Efficient High-dimensional Gradient Estimator for Stochastic Differential Equations
Authors: Shengbo Wang
Jose Blanchet
Peter Glynn
Venue: NeurIPS 2024 poster (2024)
Keywords: derivative estimation
gradient estimation
sensitivity analysis
stochastic differential equations
jump diffusions
neural SDEs
Primary area: probabilistic_methods

Abstract: Overparameterized stochastic differential equation (SDE) models have achieved remarkable success in various complex environments, such as PDE-constrained optimization, stochastic control and reinforcement learning, financial engineering, and neural SDEs. These models often feature system evolution coefficients that are parameterized by a high-dimensional vector $\theta \in \mathbb{R}^n$, aiming to optimize expectations of the SDE, such as a value function, through stochastic gradient ascent. Consequently, designing efficient gradient estimators for which the computational complexity scales well with $n$ is of significant interest. This paper introduces a novel unbiased stochastic gradient estimator—the generator gradient estimator—for which the computation time remains stable in $n$. In addition to establishing the validity of our methodology for general SDEs with jumps, we also perform numerical experiments that test our estimator in linear-quadratic control problems parameterized by high-dimensional neural networks. The results show a significant improvement in efficiency compared to the widely used pathwise differentiation method: Our estimator achieves near-constant computation times, increasingly outperforms its counterpart as $n$ increases, and does so without compromising estimation variance. These empirical findings highlight the potential of our proposed methodology for optimizing SDEs in contemporary applications.

An Efficient High-Dimensional Gradient Estimator for Stochastic Differential Equations
Shengbo Wang MS&E Stanford University Stanford, CA 94305
Jose Blanchet MS&E Stanford University
Stanford, CA 94305
shengbo.. www.@stanford.a.o.o..
jose. blanchet@stannfoooo edu
Peter Glynn MS&E
Stanford University Stanford, CA 94305
glynn@stanford. edu
Abstract
Overparameterized stochastic differential equation (SDE) models have achieved remarkable success in various complex environments, such as EDE-sonntrained optimization, stochastic control and reinforcement learning, financial engineering, and neural SDEs. These models often feature system evolution coefficients that are parameterized by high-dimensional vector IR., aiming optimize pectations of the SDE, such as a value function, through stochastic gradient ascent. Consequently, designing efficient gradient estimators for which the computational complexity scales well with n is of significant interest. This paper introduces a novel unbiased stochastic gradient estimator-the generator gradient estimator-for which the computation time remains stable in In addition to establishing the validity of our methodology for general SDEs with jumps, also perform nu- merical experiments that test our estimator in linear-quadratic control problems parameterized by high-dimensional neural networks. The results show a significant improvement in efficiency compared to the widely used pathwise differentiation method: Our estimator achieves near-constant computation times, increasingly outperforms its counterpart mation variance. These empirical findings highlight the potential of our proposed methodology for optimizing SDEs in contemporary applications.
increases, and does so without compromising esti-
Introduction
We consider a family of jump diffusions (X7(t, 8) E Idd:s E [1, that are generated by stochastic differential equations (SDEs) and indexed by the initial condition E TR at time and a parameter E C T In modern applications, the parameter 0, encoding characteristies of an engineering model, often represents the weights of deep neural network. This paper focuses particularly on scenarios where the dimension n is significantly greater than the dimension d of the space. This setting naturally arises in the implementation of large Al architectures in modern applications.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Concretely, for each 1 i d, the i'th entry of (t, denoted by X² (t,(t,)), satisfies the Itō SDE:
(t, = 2i + L. (r, X (t r))
(1.1)
J: dJe,((r) {.dJe.lr))
Σ (
Here, fpo.: : d} and for,L : 11 d} are the drift and volatility, respectively, satisfy- ing suitable regularity conditions (to be discussed). For simplicity in our introductory explanations, we will assume that the jump term Jo is zero. However, incorporating this jump feature is valuable in many applied settings, and arises in various fields such as financial engineering [17], stochastic control [6], and neural SDE models 77. Accordingly, we will fully integrate and discuss the jump components in our main results in Section
The primary objective of this paper is to develop an efficient gradient estimator, with respect to 0, for large class of path-dependent expectations derived from an SDE, Concretely, we consider
vo (1, F. // %. po(s, X (1,, (1, + go (Xz(1, TT)
(1.2)
The value (4,) represents the expected cumulative reward running X3 from time / to T. Here, Pa and go represents the reward rate and the terminal reward, respectively. This formulation encompasses a wide range of science and engineering problems including PDE-constrained optimization [22] (20) stochastic control and reinforcement learning [8), and neural SDE models (23]
The gradient (oug(t, æ> (de, vo(t, z)........
Oe_ (t x)) E R is of significant interest in the sensitivity analysis, learning, and optimization of these models. In particular, finding an efficient unbiased estimator for (t, with low variance is essential if one to apply stochastic gradient descent to find near optimal policies or model parameters within the parametric class O E O.
Under reasonable smoothness and integrability conditions, it is natural to consider the pathwise diffcrentiation estimator obtained by applying infinitesimal perturbation analysis (IPA) to the sample path of X w.r.t. the ith coordinate of 0. For instance, if p+ = p(-) independent of I and g then we have a representation
do 1o (0, = E 1
E
On pn, X² (1)d
(1.3)
where 0o X3 (t) is the pathwise derivative of the process X8 w.r.t. 6,. The processes (X 00, Xē, i 12,, 1,..., d) satisfy system of d d SDEs [T] Equation (3.31)), which must be jointly simulated. Therefore, to estimate the gradient, the pathwise differentiation method requires simulating this d d d dimensional SDE. Note that the dimension is linear in n. the dimension of the parameter space. Contemporary applications of SDEs in innforee and data-driven environments such as deep neural SDEs and deep RL where overparameterization excel, necessitate model with exceptionally large n that is often many orders of magnitude larger than d. Hence, simulating the SDE of dimension d d d- n becomes extremely noource-intensive Motivated by these applications, we ask the following question:
Can device an efficient, unbiased, and finite variance estimator for Veva(t, x) with a computation time insensitive to n??
The answer is affirmative. Precisely, our main contribution is designing the unbiased generator gradient estimator of Veug(t, z) that requires only simulating ((f)) SDEs when the volatility parameters dg do not depend on 0 and (ad)) SDEs in the general setting, summarized in Table [
We remark that in addition to pathwise differentiation, likelihood ratio-based estimators are also popular for sensitivity analysis in SDEs; see e.g. Yang and Kushner [26). However, typically they are only applicable if op independent of and under more restrictive jump structures. When applicable, likelihood ratio-based estimators could be appealing alternatives as they introduce change of measure that represents the derivatives as a functional of the d-dimensional processes Xú. Nevertheless, these estimators typically have significantly higher variance.

---

Table 1: Comparison of the dimensions of SDEs needed to be simulated.
If the volatility depends on 0
Estimator
Yes
No
Pathwise Differentiation
d d. n d d2 + d
d d
Generator Gradient
+ d
Finally, we apply our estimator to linear-quadratic control problems and test its performance in optimizing taaal--ewwor--aamettteee n, the results [igure[Taland highight a substantial improvement in computational efficiency, as compared to the pathwise differentiation method, while still maintaining competitive variance levels. Furthermore, Figure Ta confirms that the computation time of our estimator robust to increases in even in extremely high-dimensional scenarios with n approaching 10a.
controls. As we increase the number of network parameters
1.1 Literature Review
Gradient Estimation:
Gradient estimation, particularly likelihood ratios and IPA methods, is crucial
in sensitivity analysis. Foundational works in the late 20th century by Glynn [5] 4 and further adaptations to the SDE setting [26 3 highlight these developments. IPA has evolved to apply stochastic flow techniques to SDEs, both with and without reflecting boundaries [2 24 14]
Applications of Gradient Estimators: Gradient estimators widely used in stochastic control and reinforcement learning (RI.) models. Policy gradient methods in discrete-time RL., including REINFORCE and deep policy gradient approaches,
notable applications [25 [3] 21. Continuous-
time RL have been explored using policy gradients in settings with continuous diffusion dynamics (8] Jump diffusions are important models in financial engincering and stochastic control [17
[5 9 6)
Gradient estimators can also be used for optimizing these models. Neural SDE models are modern computational frameworks that model the dynamics of stochastic systems using neural-networt parameterized SDE. Chen et al. M1 Tzen and Raginsky (23). Kidger [0] focus on the continuous case, while Jia and Benson 7 consider ODEs modulated by compound Poisson jumps. Efficient gradient estimators in high-dimensional settings are crucial for fitting these SDE models.
The main technical tools for this paper are SDEs with jumps and stochastic flows. Our references are Protter 19 Kunita Øksendal and Sulem (18].
Diffusion with Jumps and Stochastic Flow:
1.2 Remarks on Paper Organizations
The paper is structured as follows: Section outlines the concepts of our estimator in a zero-jump setting, focusing on intuitive understanding over technical detail. Section [3] we introduce the SDE model with jumps and provide set of sufficient conditions that rigorously support the earlier insights. While more general and complex assumptions exist that lead to similar conclusions, these are presented in Appendix A align with the concise format of the conference proceedings. The paper concludes with Section 4, where we conduct numerical experiments on neural-network- aararaeeeoor----aaeerriiiii parameterized linear-quadratic control problems, demonstrating the effectiveness of our methodology.
2 Key Methodological Insights
In this section, we motivate our proposed generator gradient estimator by first providing a non- rigorous derivation. We assume the SDE model (I.I) where the jumps Jo and R is a bounded open neighbourhood of the origin. W.L.ogg we are interested in estimating the gradient at 0 E O and = 0: i.e. Veug x) = (0, x:) 2)
To simplify notation, we denote (1 = (0, () and (;-)))== (5 and the function
00,1)(t.x) := (t.
(2.1)
3

---

Also, for function o)) we use (, x) to denote the space derivative gradient. Similarly, do and Vo denotes the f partials.
0.v 8e 10,,2
and V the space
Under sufficient regularity conditions, by the Feynman-Kan formula, va (12) is the solution to the partial differential equation (PDE)
3ev0 + Lave + po = =0, vo = g0
(2.2)
for all 0 E θ where Lo is the generator of X given by
Lol(t = Σ
+ Σ (0,,(t,,)0,,,,(1.1
for f that is twice differentiable in Assuming enough smoothness, we formally differentiate the
PDE (2.2 w.r.t. 0, and then set 0 = 0 to obtain
0,0g, to + Lodb, + (dg, Covo + 0o, = 0, 0b (I.,) -00.00-
(2.3)
Here, the operator 0v Lo is defined as
0g. (1, = E (t, j=1
Σ i1=1
0g, 0o. (1,00,00))
(2.4)
Interpreted as the derivative of Lo w.r.t. 0 at O, this inspires the name "generator gradient" method.
Next, define 0gg.0 Treating 0g Ln as fixed, we observe that solves the PDE (2.3) which
of the form (.2).. Hence, applying the Feynman-Kac formula again to 0u (0, x)) uio (0,
yields the following expectation representation
Oo, vo x) = E '''' 0o, Covolt, x5())) + 0o (t, (f()) + 00 0o.
(2.5)
Note that the expression inside the expectation contains only space derivatives (due to 0g Co) of the value function ti but not the derivatives. In particular, we can estimate the gradient Vrn x) and the Hessian matrix H [tt] x)- == (t, :1 i,j I d) efficiently, then the representation in (2.5) will lead to a natural estimator of 0g vo (0,
To estimate (oo(L, x) and [()))(,, we employ the pathwise differentiation estimator from Specifically, under enough regularity conditions, we can interchange the derivatives and integration
Voo(t,2) x) E
Po Vgh VX.
H(vol((,2) (H(1I = E VX H + Vo H( ((TT))
(2.6)
+ E (X Hl0 + (vp, ( H[ (1,
Here, we write VXP = (0.X5, : 1,, 1,...d d} and H [ == {0,0, i, a, = 1.... d). The notation (Vh, M(x = Σa 0. (X Rdxd for h. po: -n- The dependence of Po, on time and the state process is hidden.
We estimate these expectations by simulating the SDEs for (X3, VXY. H[x]] given by
I.1 and
0ax%, 8r.a + f.¹ Σ
S
1-
0,0,X0,5 =
Σ
0rpo. + Σ
Om, 05 dr
(2.7)
1=1
m=l
0,00,,,0,00,53 Σ
b Um000oss44x77,00000 dB&(r)

---

where the dependence of the coefficients on (t, and 2, as well as the dependence of Xi. 0, 0,0, on (t. s). (t. rt.r)) are suppressed.
The dimension of these SDEs d + d² + 2 where the 2 comes from the Hessian being symmetric. Moreover, when the volatility is independent of 0, method only necessitates estimating Vig- This reduction leads to simulating the SDEs for {X6,, VXū of dimension only d + d²
Assuming sufficient integrabilityy, the unbiasedness of Z implies
0o Ho (6,, t))dt E [ Ooelo (0,
E
(2.8)
which we will elaborate upon in (A.1 The same holds for the H(t,x) x) process well. Therefore, we can replace the derivatives Veg with Z and [vo] with H (.. without changing the expectation.
Also note that producing a sample of (tt, x) requires simulating the solution to SDEs (I.1 and (2.7)
within time [t. I'] starting from x, 1, So, it t; a similar issue exists for H as well. This can be addressed by randomizing the integral.
very efficient to compute Z(t, X6 (0, for every
With these considerations, we proceed to define the generator gradient estimator. First, let V V Lo (o x) be defined by replacing 0,0(t, æ) with (t, x) and 0,0,v with Ht,((t, x) in the defini- tion 2.4) of 0s, Lovolt, 2).. Then, define the generator gradient estimator
L'' Vapo(t, + (X))
D(x) = 1VoLLo0(,,
(2.9)
where T On T]
sampled independently. We can also randomize the integral of Vapo (t)) if the gradient is hard to compute. With the derivation in (.8). it is easy to see that ED(x) (0vn(O, x) is unbiased.
In summary, due to the observation in (.55). we are able to "move" the estimation of V V onto that of Vog and [[o]]. This results in significant reduction in the dimension of the SDEs we need to simulate, underlying the remarkable efficiency of our methodology, especially when the dimension n of @ significantly exceeds d.
3 Jump Diffusions and the Generator Gradient Estimator
In this section, we rigorously formulate jump diffusion process driven by SDE. We extend the generator gradient estimator to this context by first rigorously establishing an expectation rep- resentation of the derivative in 2.5). Then, we also validate the representation (2.6) using the jump version of (2.7). These lead to our generator gradient estimator in the jump diffusion context. will state set of sufficient assumptions that are easy to verify. However, we will state and prove our theorems using a set of more general assumptions in the Appendix A
To improve the clarity of the paper (at cost of generalizability).
We consider jump diffusions on the canonical probability space of cådlàg functions [0, T] Rd generated by SDEs of the form [.I where the jump term is given by
(t, xz J." [ (x::6.)X6((t))) S Σ
(3.1)
+ X.
xo.ll, (8, (s) 2)dN dz).
In this expression, B is a standard Brownian motion in RR'' Ñ is compensated Poisson random measure with intensity measure dt x v(dz) with v a Lévy measure on (Rg := Rd \ {0} ((R )). i.c. Jag A (z|2v(dz) oo; the -r notation in (t, denotes the left limit; and the stochastic integrations are Itô integrals. Here, for vector/matrix//ensonn
xddxxdxxd, Eigh [Vi.,e|2 1 We further define 7>)) = |z| A 1 and y(dz) = (dz). Then is a finite
denote |o|2 :=
measure on (TRg) B(RR)))) )). Also, since we are interested in the gradient at = O, we can assume w.l.ogg. that θ is bounded open neighbourhood of 0.
5

---

The generator of this system of SDEs is La: := C + Cij, where
Cg ftt, = Σ Me (e, z)).ftz) +
Σ
a0 (t, (t,,),,;f((t.))
+,==1
(3.2)
C f(t, = =J ag
-[t,x- - f(t, x)
d Σ 2,) x)) v(dz).
for E ((O, (P,(O,,).R) TJ.R). We remark that for open subsets W, X, the space C* 7), W X) represents the set of functions / on [o, T) x W that has continuous mixed partial derivatives 0r 0 f on (0, 7) x W for every a S i,b S S k. Moreover, these mixed partial derivatives
have continuous extensions on [0, T] x W x X.
3.1 Probabilistic Representation of the Gradient
In this section, rigorously establish the probabilistic representation of the gradient Vav (0, as outlined in equation (2.5). Our approach leverages the continuous dependence of 0 + X of
the solutions to (3.T) in neighbourhood of 0, given sufficient regularity conditions. This behavior extends the properties associated with stochastic flows, as explained in the work by Kunita [T.
Recall that @ is a bounded neighbourhood of O R" To clarify the assumptions, we enlarge O and consider O = {0 + v E G, v E (*(0, })} where (0,, e) is the open ball in R" at 0 of radius e. Assumption 1. For some e O, the following regularity conditions hold
1. The mappings (s, 0, + (o(s, a8(8 x), Po(8, 9o x) are C0.1.1 For each z E Rg", (s, 0,) - xo(s, 2)/((2) is C (IO, T7. D..RR) Moreover, [xo 2)))(()))((( is uniformly bounded in [0, and E Rg
2. The spacial derivatives Pul. /ool and | Xxo are uniformly bounded. The 0 derivatives satisfy linear growth
[opo(s, (o Voxo(s, (22)
[(([=] + 1)
for all s E [0,T), x E Rd, 2 E Rd and e.
3. The derivatives of the rewards satisfy polynomial growth: for some m 2 1,
Vepo(s, x) + Vege(z)] S E([x] 1)
for all =00,T], I E Rd, and 0 E O.
Remark. Requirement 1 implies that for each fixed x, the 0 derivatives of the coefficients are uniformly bounded in [0, 7 + as θ is assumed to be bounded. So, the seemingly strong requirements of the derivative satisfying the growth condition in items 2 and 3 are not very restrictive. The boundedness of Xe (s, x. (2))(2) in is relaxed in Assumption 5 in the appendix, allowing unbounded jumps. The strong condition is the uniform boundedness of Vool /ooll and | X However, this is typically necessary for the existence and uniqueness of strong solutions the SDE (3.1).
Q are classical solutions to the partial- Assumption 2. Assume that {vg E C ((0, Rd)
O E
integro-differential equations (PIDE)
0.00 + Lave + PO = 0, vo (o 9o
where La = LG. + Lif are defined in 3.2). Moreover, up and its space derivatives satisfy polynomial growth: for each O, there exists Co < Do and m 2 1 s.t.
|vo x)) (|r) 1)m
V (t,
H Huvo (vo](tt,))) co. (|0) (E
sup =ER-,EE(O,T]
< ca.
sup vERd,cEOO.T
(|0) + 1)mh
co.
sup
1)mm
Remark. By classical solution, we mean that vo satisfies 0.vp- + Love Pa = 0 on (0, 7) x Rd with its continuous extensions of satisfying (T, = 9o- This possible, for example, in settings with
2 terminal rewards. Note that is a stronger requirement compared to the definition in Evans (2].

---

As we have motivated in Section 2 Assumption follows
ffllmt from generalized version of the Feynman- Kac formula, under additional technical assumptions. Moreover, the growth of Up and its space derivatives can be derived from assumptions on the growth of the rewards. However, in order to not obscure the main message the paper and to streamline the proof, we directly assume these properties. We refer interested readers to Kunita [II] Chapter 4] where stochastic flow techniques similar to the proofs the paper are employed to establish the PIDE and validate the growth rates.
Theorem (Probabilistic Representation of the Gradient). If Asssmmttinss 7 and 2 are in force, then 0 vo(0, x) is differentiable at O. Moreover, the gradient
( ( (opo(((()) +Voo (ogo(X)((T))
Vo)n(0,x))
where VoLo := Voly VoLl s.I. for f(1,x)) E C
d V Volo f(t,z) Σ
V (oHo,(t.
+ Σ (t),,,)
(3.3)
1
voC/f(t,x))
Σ
-. v(dz).
(3.4)
In Theorem I we have successfully established an expectation representation of the gradient Vova (0, x:) of the form (2.5). This naturally leads to the consideration of using Monte Carlo to estimate (O, r). However, one observes that the representation Theorem involves the space derivatives 0,00 a:) and 0,0, which are usually hard to compute exactly.
In the next section section, following the heuristics in (2.6) we establish conditions on the model primitives so that the space derivatives 0,v0 (t, x) and 0, oo (f, z) admit probabilistic representations as expectations of random processes {Xo, V VX.. [[]]] that can be easily simulated.
3.2 Probabilistic Representation of the Space Derivatives
We proceed with introducing assumptions that guarantee Theorem 2 providing representations of 0,v z) and 0o .)) as illustrated in (.6). To achieve this, we first need to ensure that the derivative of the mapping x > X is well defined. This formally established Proposition A.I
Assumption 3. For each = E Reg', the SDE coesficients (s,2) (00(,)) Xols, x 2)) are ( 11, ) For each i,j I....,d, the coefficients and derivatives, (8, x) + (a(s,))) x). (),, x()) where (ax, B,C) (Ho, oo. Xo/2), (0,po, 0.00, dixo/T), and (0,0, 0,0,0u, 0,0,xu/7) are uniformly Lipschitz; i.e. there exists 0 I l < 20 s.s. for all E [0, [), Rd
functions
(a(8,2) x0)) [B(s, x)) ((s, (( ((8,,', S slpx x'1.
Moreover, ((s,0,,)) is uniformly bounded for s [0,T] and z E Ing"
In view of this assumption, consider the following SDEs, jump versions of 2.7p. for which the strong solutions should be the space derivatives of Xg. Again, the dependence of the coefficients on X (t, r-), and as well the dependence of X 0,Xo 0,0, on (t, s), (t, r-) has been
7

---

suppressed.
J d
J
0,x òūaa +
da
+
1.
0.0,X6,5
Σ
Orp0, + Σ
Om 0m
(3.5)
1=1
m=1
d'
+
0, +
Σ
00 dBa(r)
m=1
θ 0, +
Σ
dN dz).
As will show in Proposition A.1 twice continuously differentiable in æ for every S t < s < T. The processes (VXg. H [[]]). as defined in (3.5). will then correspond to the derivatives. Moreover, these processes, well as Xj. will possess desirable integrability properties.
under Assumption 3 the process (t, s) has version that is
To guarantee sufficient integrability and to provide a variance bound for our estimator, we also need assume growth conditions on the rewards.
Assumption 4. Assume that the mapping x po(1,22), is C2 for all [0, T. Moreover, for k(1,x) = po(l, x)) and go(x) there exists Cp s.1.
[h(t,x)]
Vh((,
[t,,(t, (|æ) + 1)TH
sup (E (|æ) + 1)mh
I Ck.
sup (E,
I ch.
sup #ERM (t(00
S Ch.
(|0) + 1)m
With these assumptions, we validate the representations in 2.6 using the following theorem.
Theorem 2 (Probabilistic Representation of the Space Derivatives). Under Assumptions 3 and 4 the
representations in (2.6) hold with the jump version of (X6, VXy H[x]]] in (3.1 and (3.5)
3.3 The Generator Gradient Estimator
With Theorems T and 2 we construct our generator gradient estimator and show that is unbiased
with variance that grows polynomially in Recall the estimators Z(t, and HI 2) in (2.6).
By Theorem 2 and the integrability Proposition A.l under Assumption 3 the equality (28) holds.
Then, following the notation in (2.9, we define
V Lo = VoLO + (oI'Vo(tt,
where VoLSVOLL r) and V Lg Voll, x)) are defined by replacing 0,v(1, x) with Zz x) and 0,0,v with Hi,s(! x)) (3.3) and (3.4. respectively. Then, our estimator D(x) is given by (2.
Theorem 3. Suppose Assumptions 1-4 are in force. Then, the generator grudient estimator D(x) is
unbiased; i.e. ED(x)) V Vo,0(0,z) x). Moreover, the variance Var(D(s)) C(x=| + 1)2m+4 has at most polynomial growth in z, where the constant C can be dependent other parameters of the problem but not z.
Remark. The signifies the growth rate of the rewards and their derivatives. The extra additive factor 2 in the variance is from the growth of the 0 derivative of og, the volatility squared.
4 Example: Linear System with Quadratic Loss
In this section, we illustrate some analytical properties and the effectiveness of our estimator by considering a linear quadratic control problem.

---

25
Generator Gradient
5000
Geberator Gradient
Pathwise Differentiation
95% CT
20
Estimated value
4000
Pathwise Differentiation
Avg runtime
15
10
1000
-1000
10a
10a
10"
10"
10F
102
10
20
40
60
100
Dimension of
Coordinate index of
(a) Average runtime for different n.
(b) Estimator statisties with 102.
Figure I: Comparisons of 100-sample estimation statistics and averaged runtime.
Let X E Rd be the controlled process, given by the solution to the SDE
x²t)== = x / (x* (U + L: CdB(s),
where Bt) E Rdd is standard Brownian motion, (t) E R the control process that adapted to the filtration generated by X, A E RRdxd B E Rdxm C Rdxd' are non-random matrices. The objective is to choose an admissible control (t) that minimizes the quadratic loss
E [
X (1) U()) x X'T)
where Q, Qr E Rdd and R E mm are non-random matrices.
In various applications of interests, the admissible control (tt) is parameterized function of time and state (t)) = ug X5 (t)) where the state process under control uy is denoted by Xş. The dimension of could potentially be very high-e.g. when no is neural network. To achieve optimized loss in this over-parvee peee-parametrized setting, one common approach to run gradient descent. Hence, efficient gradient estimator that scales well with the dimension of is highly desirable.
We compare the performance of the proposed generator gradient estimator and the pathwise differen- tiation estimator. In this context, these estimators take the following form. The detailed derivations are presented Appendix F.TT
The Generator Gradient Estimator: In this setting, our generator gradient estimator in 2.9 is
D,(x) T0g,ue(T: ( B Z(r, (z()) - Tuo (T(T)) (R
R RT00, uo X²(T))
where the definition of Z follows from (2.).. and is given by (F.1 in Appendix F.1 As explained in we also randomize the integral corresponding to the gradient the reward rate Vopo.
2.9
The Pathwise Differentiation Estimator:
From
1.3 we find the following IPA estimator that
randomizes the time integral
5,(z) = Tup(T, (T) R. ( - 00,
+ (T) HR+RRT uo (T))
+ (T) (T. (4r+
( X7
Here, the pathwise derivatives 0g, (t) is the solution to
We deploy these estimators in an environment where the state variable r R represents the x-y positions and velocities of a point mass on 2D plane. The controller applies force to this mass. The cost function designed encourage the controller to swiftly move the point mass to the origin with minimal force. The force is sate-timmedeeeedenn and parameterized through 4-layer fully connected neural network with variable width. All computation times are recorded from Tesla V100 GPU. Further details about the setup of our numerical experiments can be found in Appendix F2

---

In Figure Fa we present comparison of the average runtime for computing a single sample of the generator gradient and the pathwise differentiation estimators D(z), D(x) I2", across increasing values of n the dimension of Our findings indicate that the generator gradient estimator not only outperforms the widely used pathwise differentiation method across all tested values of but also surpasses it by more than an order of magnitude for larger values of n. Additionally, the computation time for our estimator shows remarkable stability with respect to increases in displaying only slight uptrend when n Z 10".
Figure Tb confirms that, at = 102, the estimated values by the two estimators are very similar with high confidence. This confirms that our estimator consistently estimating the gradient (o, z.).
Table 2: sampl standard error (SE) comparison between generator gradient (GG) and pathwise differentiation (PD) estimators.
(dimension of ()
102
1002
5502
21002
3.24e5
1.29e6
5.14e6
1.15e7
Avg SE of GG
5.253
5.785
3.533
1.205
0.965
0.729
0.600
0.407
Avg SE of PD
6.424
5.710
4.453
1.191
1.110
0.935
0.786
0.466
Avg SE ratios
0.971
0.932
0.946
0.903
0.902
0.914
0.926
0.961
Finally, Table presents the standard errors (SE) (F.4) from 400 replications of both estimators, averaged over the gradient coordinates. It also displayed the averaged ratios of the standard errors F55). We observe averaged SE ratios that are consistently less than for all n, suggesting that our shown in Figure La but also achieves lower estimation variances. Further analysis the SEs for each gradient coordinate is conducted and displayed in Figure 2 in Appendix F22, highlighting similar histogram shapes and observable reduction in large values of SEs of our estimator.
generator gradient estimator not only provides significantly faster computations
Concluding Remarks
The theoretical results in this paper have the limitation of requiring second-order continuous differ- entiability and uniform boundedness of the space derivatives of the parameters of the underlying jump diffusion. These strong conditions, which are standard the literature of stochastic flows (cf. [1) to guarantee global existence and uniqueness of the derivative processes in (3.5). are necessary to achieve the generality of the results presented in this paper.
However, our generator gradient estimator often works even when coefficients are not continuously differentiable. This is true if the generator and rewards gradients defined almost everywhere, and the derivative processes in (3.5). with almost everywhere derivatives of the SDE parameters, exist for every I [0, T) and satisfy some integrability conditions. Examples include neural networks parameterized stochastic control with ReL.U activation functions, heavy-traffic limits of controlled multi-server queues, and the Cox-Ingersoll- (CIR) model. For these models, the existence and integrability of the derivative processes can be checked on case-by-case basis, allowing the consistency and unbiasedness of the generator gradient estimator to be established. We confirm this by numerically investigating the CIR process and an SDE with ReLU drift in Appendix G
Acknowledgments and Disclosure of Funding
The material in this paper is based upon work supported by the Air Force Office of Scientific Research under award number 11550020-1-0397 Additional support is gratefully acknowledged from NSF 2118199, 2229012, 22044, and ONR 11983111.
References
[1] R. Chen, Y. Rubanova, J. Bettencourt, and Duvenaud. Neural ordinary differential equations. Advances in Neural Information Processing Systems, 31, 2018
[2] L. C. Evans. Partial Differential Equations, volume 19, American Mathematical Society, 2022.
10

---

[3] W. Fang and M. B. Giles. Importance sampling for pathwise sensitivity of stochastic chaotic systems. SIAM/ASA Journal Uncertainty Quantification, 9(3):121711241, 2021.
[4] W. Glynn. Optimization of stochastic systems via simulation. In Proceedings of the 2 Winter Simulation Conference, pages 90-105, 1989.
[5] W. Glyun. Likelihood ratio gradient estimation for stochastic systems. Communications of the ACM, 33(10):75-84, 1990.
[6] X. Guo, A. Hu, and Y. Zhang. Reinforcement learning for linear-convex models with jumps via stability analysis of feedback controls. SIAM Journal on Control and Optimization, 61(2): 755-787, 2023.
[7] Jia and A. R. Benson, Neural jump stochastic differential equations. Advances in Neural Information Processing Systems, 32, 2019.
[8] Y. Jia and X. Y. Zhou. Policy gradient and actor-critic learning in continuous time and space: Theory and algorithms. Journal of Machine Learning Research, 23(275):1-50, 2022.
[9] I. Karatzas, S. E. Shreve, 1. Karatzas, and S. E. Shreve. Methods of Mathematical Finance, volume 39. Springer, 1998.
[10] P. Kidger. On neural differential equations. arXiv preprint arXiv:2202.02433 2022.
[11] H. Kunita. Stochastic Flows and
Springer, 2019.
[12] X. Li, T.KK. L. Wong, differential equations. In S. Chiappa and R. Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages
Q. Chen, and D. Duvenaud. Scalable gradients for stochastic
PMLR, 26-28 Aug 2020.
[13] T. P. Lillierap, J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint
2015.
[14] D. Lipshutz and K. Ramanan. A Monte Carlo method for estimating sensitivities of reflected diffusions in convex polyhedral domains. Stochastic Systems, 92):1011-40, 2019.
[15] D. B. Madan, P. Carr, and E. C. Chang. The variance gamma process and option pricing. Review of Finance, 2(1):79-105, 1998.
[16] Massaroli, M. Poli, Peluchetti, J. Park, A. Yamashita, and H. Asama. Learning stochastic optimal policies via gradient descent. IEEE Control Systems Letters, 6:1094-1099, 2021.
[17] R. C. Merton. Option pricing when underlying stock returns are discontinuous. Journal of Financial Economics, 3(1):125-144, 1976. ISSN 0304-405X.
[18] B. Øksendal and A. Sulem. Applied Stochastic Control of Jump Diffusions. Springer, 2019.
[19] Protter. Stochastic Integration and Differential Equations. Springer-Verlag. Berlin, Heidelberg. second edition, 1992.
[20] J. Sirignano, J. MacArt, and K. Spiliopoulos. Pde-constrained models with neural network terms: Optimization and global convergence. Journal of Computational Physics, 481:1 12016, 2023.
[21] R. S. Sutton, McAllester, S. Singh, and Mansour. Policy gradient methods for rein- forcement learning with function approximation. Advances in Neural Information Processing Systems, 12, 1999.
[22] Tröltzsch. Optimal Control of Partial Differential Equations: Theory, Methods, and Applica- fions, volume 112. American Mathematical Soc., 2010.
[23] B. Tzen and M. Raginsky. Neural stochastic differential equations: deep latent Gaussian models in the diffusion limit. arXiv preprint 190.vv:100.000000 2019.

---

[24] Z. Wang and J. Sirignano. A forward propagation algorithm for online optimization of nonlinear stochastic differential equations. arXiv preprint arXiv:2207.04496 2022.
[25] R. J. Williams. Simple statistical gradient-followinn algorithms for connectionist reinforcement learning. Machine Learning, 8:229-256, 1992.
[26] J. Yang and H. J. Kushner. A Monte Carlo method for sensitivity analysis and parametric optimization of nonlinear stochastic systems. SIAM Journal on Control and Optimization, 29 (5):1216-129,, 1991.
12

---

Appendices
A Generalizations of the Assumptions
A.1 Probabilistic Representation of the Gradient
In this section, we develop a generalized version of Theorem 1 weakening Assumptions 1 and 2 In particular, we allow discontinuities in time of the SDE coefficients. This flexibility is especially relevant in applications in data-driven decision-making environments where non-homogeneous SDE models with estimated drift, volatility, and jump parameters could be piece-wise constant. Moreover, we also relax the differentiability of the coefficients in the space variable to Lipschitz continuity, We will state the new set of assumptions, and establish generalized version of Theorem ass in Theorem
We proceed by presenting critical theorem, along with the necessary assumptions, that forms the foundation of our probabilistic representation in Theorem T
Assumption 5. Assume that for each 0 the coefficients pe(;,)), oe(..)), and xo(:...) are jointly Borel measurable. Moreover, assume the following holds true:
1. At x 0, the coefficients are bounded: for all 2
/
xa(ss
P
[ua(s, 0]1 (o, 0)1 (o0,s))
u(dz) < 00
sup (E
Rg
7(2)
2. The coefficients are uniform Lipschitz in x, uniformly in s,0 in the following sense: there exists constants and {cp p 2 2} s.t.
[po(s, x) - po x' S c | 2'1, loo(s, -00,))))ooos S c/z 2'1,
and for any 2
1/
x0(s,
xo(s, 7(2)
u(dz) S cpll x'l
7(2)
for all [0,T], x, E Rd, 0 A.
3. The coefficients are weakly Lipschitz in 0 the following sense: for each 2 2, there exists a time dependenn positive field { 1il (() E ERso: :s: [0, 0' E 0 s.t. for some constant Ep.
(L s 10 0'
for all 0, O' E O and the coefficients satisfy
(oo x) (8, < (1) l00(s, - (0) S (s)
and
x0(5,
X (s, 72)
P u(dz) S 12 (s) + 1)P
Jng'
(2)
for all E [O, 0 E θ, E Rd. Theorem K (Theorem 3.3.1 of Kunita
11). If Assumption 5 in force, then (Xz(t) it E [0, 0 E @)} has a version X (i.e. C N with P(N) = 0) that B([0, T)) B(O) x F mea- surable. Moreover. 1, for each 0 E O. (w, t) is cadlag in t, and A + (w. seen as mapping θ + (D(0, T] I1 lloo) is uniformly continuous on compacts. Furthermore, for any p 2 there exists by (0) s.t.
the family of solutions X ( ( Xg) * xg
sup E sup (t) < 65[(,r + 1)P
dee (E(O.T)
13

---

Remark. Theorem K is an extension of Kunita [1] Theorem 3.3.1] using the a.s. version of Kolmogorov's 's continuity criterion; see Corollary 1 of Protter 19 Theorem 73].
To guarantee that Theorem I holds, the requirement that Assumption 5 for all p 2 2 can be relaxed to all p theory focuses on regime where n >> d. So, we adopted this stronger version of Assumption 5 This also clarifies the presentations of the following assumptions: To guarantee the main results of this paper, a weaker version of this assumption requires that the assumption holds for all S 4m + where the growth rate of U, r. 9, and their derivatives in Assumption g and
where is the dimension of f. However, the intended application of
Next, we present additional regularities that implies the probabilistic representation in Theorem Assumption 6. For each E [0,T] and (8, x), oo (8, x), X0(8, po, (), (0, x) are (1.((O,R4)
R the mappings (0,
Assumption 7. The measurable reward rate po and terminal reward go functions are Lipschitz in in the following sense:
1. There exist m 2 a > 1, and 1 (8) CRso E [0, E 6 s.1.
"/ for all 0, 0' E O and the reward rate satisfies
Sla l1
lpa 2) per (s, 2)1" S NG (0 + 1)ma
for all [O. 0,0' e Rd
2. For some 2 the terminal reward satisfies
(go(x)) go (x) I 0 8'((((1 + 1)mh
for all [0, 0, O, Rd
Note that for notation simplicity, w.l.o.g. we use the same Kō,er and la as in part of Assumption denote the Lipschitz coefficient, and the same m as in Assumption 9
Remark. is not hard to see that Assumptions 6 along with and are generalization of Assumption
I i.. if Assumption I holds then will Assumptions 6 and
Next, we slightly generalize the growth part in Assumption 2 as in Assumption 9
Assumption Assume that {vo E ((o, )
: 0 E θ } is family of classical solution to the
PIDEs
0wyg + Lovo po
(T, = ga
where Cs = CC + L.j as defined in 3.2
Assumption 9. There exists Co < oo and m 2 s.t.
|vo(t, x)
[Vn x)
H (vol(tt, x) ((rl + 1)m
sup
I Cu.
sup x037,1
S Cv. and
sup tERd)E((077]
S Co.
=CR(T (Jxr + 1)m
(arl + 1)m
Moreover, for each 0, there exists ca.v s.f.
[o)(t,
sup #ERM (o) (|z) + 1)mh S CØ.v
Theorem P.. If Asssmpptisss
and 9 are in force, then the statement in Theorem 7
hold; i.e.
/
V Lo V Vagu (xgo(XX((T) (0, E
where VoLo VoLe VeLy are define in
3.3 and
3.4 respectively.
14

---

A.2 Probabilistic Representation of the Space Derivatives
Following the same spirit, in this section, we develop a generalized version of Theorem weakening the Assumption B to the following Assumption:
Assumption 10. For each E [0. TT E Rg', the coefficients x - (na((,, x), co(s, , x))) =))
second continuously differentiable. Moreover for each i.j 1,...,d, the coefficients and derivatives, seen as functions (8,z) - (a(s,,)) (), ((,,x,)) where (ax, 3,C) = (No, 0o Xo). (O 0, 0.Xo). and (3,0,1o 0,0,00: 0,00x))))))))) satisfies the following conditions:
1. At 0, the coefficients are uniformly bounded in time: for all p 2
((s,0,2) (22)
sup sE(U,T]
la(s, +) + 18(,,)) +
u(dz) <0
2. The coefficients are uniform Lipschitz in x: There exists constants e and (cp : p 2 2} s.t. (o((s x) a(s, S cel I'1, 18(8,z) B(s,,')) scl
and for any 2 2
((8,01,) 7(2)
((8,,',2) u(dz))
S cpll x'l
(2)
for all E [0, x, E Rdd
Proposition A.1. Suppose that Assumption 10 is in force. Then, the family of stochastic flow solutions (x6 t),0 < S t T : x E Rd) of the SDEs 3.1 has version Xo that is second differentiable in at any time. Moreover, { (8g VX H[X x E Rd) is version of the solutions of the systems
of SDES (3.T and 3.5). Further, the family of solutions of 3.1 and 3.5) satisfies the following properties:
1. For each p 2 1. there is 0 < by 60 S.t.
E sup tE|O,T]
[((0)*=$$(10 + 1)2
and the derivatives
sup E sup #ERd 05ASIST
(x S bbp p
sup E sup
(s) s br.
*ER 03eSi<T
2. For any 21 1, there exists 0 < 1p < xc s.t. for all x E Rdd,
E sup 0S+SIST
(s, - X.6 (s $ 1n x x x'12²
E sup (s, VX," t) < 1/1x z'P.. OSASIST
A proof of Proposition A. is provided in Appendix C
Remark. We observe that part 2 of Assumption TO will imply the space derivatives of the coefficients bounded, which used get the I.P boundedness and Lipschitzness of the derivative processes. Assumption TO also ensures that the second derivative is uniformly Lipschitz as well. This is not used in the proof for the upcoming results.
We establish the following Theorem 2 generalizing 2 The proof is deferred to Appendix D
Theorem 2'. Under Assumptions
10 and then (2.6 holds; i.e.
Voo(t,x)T= = E
V (t, V Vx
H[vb](t,x) = E
VX [
dr
+ E (x H VX.
go. H [[ ((t, T)
where (X6, VXg, {[x]] are the strong solutions to
and
3.5
15

---

A.3 The Estimator
With Proposition A.1 and Theorem we are ready define our generator gradient estimator for (0 (0,z) z) and show that it is unbiased and has variance that grows polynomially in 2.
First, recall the definition of Z and H in (.6
z(t,z) ' Po VX) Vgl
H(1,)) = ( ( dr
(t,
(Xx,
Observe that by Theorem the integrability in Proposition A.I
E
Oo, (f Z(t, (0,
[" diE (1,, E [" Veb Vxx X6000) (1r))r Val V 16 (1, (g(),))
E I Mol (e,
Veo X (0, t) d):
(A.1)
similar property hold for the (t, process as well. Therefore, we can replace the derivatives Vup with Z and H [P]] with H in Theorem without changing the expectation, showing the validity of particular, this implies that the generator gradient estimator defined in (2.9) unbiased.
2.8
Finally, we establish a generalized version Theorem 3 using the assumptions in this section. The additional proof of this theorem is presented in Appendix E
Theorem 3'. Suppose Assumptions 4-10 are in force. Then, the generator gradient estimator (2. is unbiased; i.e. ED(x) Veva 0,, x) Moreover, if the
in item 1 of Assumption replaced by 2, then the variance Var(D(x)) S C([x] + 1)2m++44 has at most polynomial growth in x, where C' can be dependent Remark. The m signifies the growth rate of the rewards and their derivatives. The extra additive factor 2 in the variance is from the growth of the f derivative of og, the volatility squared.
other parameters the problem but not
B Proof of Theorem
In this section, we prove Theorem I and hence the simplified Theorem I
Proof of Theorem
First, we recall Ito's formula. For f C²222(00 x Rd).
d ΣΣ 0,f(t, df (t(())))) = [0.f (1,f(t (C) dt + [ if(t, X((t, + Xo -(t (dt,,
+ (C)
f(t,
(dt, dz),
(B.1)
where the operators CJ and LJ are defined in (3.
Then, an application of Ito's formula (B. under Assumption g and yields the following result for which the proof is presented in Section B.I
Lemma 1. For any E θ
MI,o(1) = vo(1, (1)) (0) / pa(s, (z))da
Mo,o(t) = uo Xi (t)) (0, f"' 0,vols, (s)) + Lovo(s, (0) ds))
martingales for TT
16

---

Therefore,
0 = L (T))
Then, rearranging terms, one gets
2) - -00)),00000 (0,
())((IX)(TT)) (0)())))~((TTX80 (o /
0..0 + XX (s)))d
Lavo(s, @ Ego - -xx))))))))(())) [' po(s, - X)(s)) Cavo, (²()) - Lovols, (s))dss
E (cg - co) vo X3 1' (C) -i)) (7
Ego - (T() / po X²(s)) polss X1 (s))ds
(B.2) where (i) follows from Assumption 8 that (T. go(-) and (o -0, (s, for all x E RRd and 0 T, and (ti) recalls the definition that La = Lg + Lig.
To conclude Theorem 1 we analyze the finite difference approximations of the above three expecta- tions, where they correspond to the continuous, the jump, and the rewards part, respectively. The results are summarized by the following Proposition B.1 whose proof deferred Appendix B.2
Proposition B.1. Under the assumptions of Theorem 7 for K C,J,
1 0
E SN
lim R0
(ch -6)) vo(s,
ot E
where Voly and Veli are defined in 3.3 and
3.4 respectively, Moreover,
T
0² E 1 Vepo(s,Xo000 (x()))ds
lit
E
po(s,
po(s. X5 (s))ds
and
lim 8-0
E. [ga (x (x( - /r FV (X)(T)) =
With Proposition B. handling each term in B.2 we conclude that
06 00 x) v0 D² % VoLov((s,X((6)) + (opo(X)(s))ds + Vego(X
lim 0 0-+
=0
0
B.1 Proof of Lemmall
Apply Itô's formula
B.I vo (t(t)) yield
= (t, Xx (t)) - v0, [ (oul(tXX(s)) - Lo X² (s)
d d'
O,aa (o
L'' [ (vo(s, X (3-) xo Xx - 10 X3 ( (s-)) di (ds, dz)
Jm
=: Ma, - (t) -t())
17

---

by Assumption 8 where (t) and (t) denotes the Itō stochastic integrals on the previous lines, respectively. Since the integrands are a.s. finite, (t) and (t) are local martingales. We show that they are true martingales. First, for 7, apply the Dariolddee---vii-e-eee inequality
E ( < P sup (1)
SS
E
ds
<CB
(u (s)) [od ds
Xy EsE / (oo(s) oo(s, oo(()))) os(, (0( ds
CE
where C/ is some constant that could change line by line. Notice that by Assumption 5 (t, =: By < co.
sup
(B.3)
Therefore, by Assumption 3 item 2, Assumption and Theorem
E)1(P S CE ['' (+) + -> E&E L
00.
Therefore, In is a martingale. For 12, by Kunita
Proposition 2.6.1]
f JRd
0o (s) + (o Xg(s), 2)) up Xx X((-)) u(dz)ds
( < CE
7(2)
1 CE S""
Vuo(s, Xz(s) Exo(s, X3 (s), 2)) xo (3()) u(dz)ds
JRy
7(2)
(s(s, (z(s). 2) (=)
p(dz)ds E 5 / (uu Exo Xf(), u(dz)ds
CE
Rg
Rd
where (i) follows from the mean value theorem with & == Eo(s, (x), z) E [0. 1]. and (ii) follows from the Cauchy-Schwartz inequality applied to the integral w.r.t. the finite measure P x Teb n. By Assumption 5
(x(ss,0 0, 2) 7(2)
X@.v = sup SE 0,21 Rd"
u(dz) < oo
Again, by Assumption 5 item 2 and Theorem
(o(s, 2s)
m(dz)ds < CE 1 Rdd
Xo(s, (s),, - -xo(s,022
E
u(dz)ds + CXi.v
0
7(2)
(2))
< xē.v + E [xx(s))(ds
< 00.
Also, by Assumption 9 and & E [0,
E
V ooo X3()) + Exals (z)), uddz)ds
I E 1'd
/ sg
(x) + Ixo Xx (s), 2)) + +1) 1) u(dz)ds
E 1/' [rg (x)s,, 4m
5 C
E
(m)
7(2) 1 u(d2)ds
Jo
< 0
18

---

where (i) follows from ze being finite measure and 7(2) = | A < 1. This shows that Iz, hence (o. martingale.
To show that (t) is martingale, we employ the same derivation with ey replaced by uo. This completes the proof of Lemma
B.2 Proof of Proposition B.1
Since X is indistinguishable from X=, we can use X and X interchangeably when evaluating expectations. Therefore, it is understood that we use X= when we need continuity in 8, while we keep the notation X².
In this proof, for notation simplicity, the letter C' will denote a constant that could change from line to line. C can be dependent the dimensions d, d', the growth rate m, the horizon T, the Lévy measure v and polynomial power p or a. But it doesn't depend on & (or sometimes d) and x.
The Continuous Part: We prove the claim that the derivative for the continuous part should be
E
VoLy vnls (f)))ds,
(B.4)
where VoLu is defined in (33 To proceed, also claim that
E L/ (c. 1 -8) vo(s, (F(s)ds < <c, E
L/ 1os, <
(B.5)
so that the derivative ratio and the derivative are well-defined. The finiteness of these expectations is shown below.
To prove the claimed expression B.4) is indeed the derivative, we consider the limit
%0 (c) =g) vo(s 0T J. VoLy (s.
lim 0
(B.6)
< T lim E 1
|0 (cg vol vols, ((33 0t VoLo 0o (z)) ds
T
To show the r.h.s. go to 0, first show what's inside the two integrals is U.I. Consider for
1
E
vo(s) - (4 ds
T
J
10
1
T
1
1 [Cg Lg) vo(s, X7(s)) 1 E / VoLy X(())) ds
< 2
E
|0
(B.7)
For the first term, consider
Icg - -8) (t, x)
(molt,,r) 1o (t) V (1) + Σ (t, an.., (10000))))
1,j=1
(B.8)
< (I0((t.)) (o(1, x) (.,n((, + lao (1, ao (1, (t,
(
S co(l + 1)th (lua (t(t lao(t, x) oo(t,))))
19

---

where (1) follows from Assumption 9
For the second term, recall the definition in 2.1
(a)(t, - =(o(tx))
ao (t, (t,xx
sd=1
Σ go 00)a(l.))00ssMl..0 -
1
Σ
Σ
- (oss o0,, + + to - (t,,
2 i
d
d'
10,, x)
+ loo.alt,x)loooostttt (ooe:///oloolaoooooooooooo/o//ar ot,)lttx))
2
The terms can be handled in the same way follows:
(0,, - Go. -ou (005,,((tt
k===1k=1 km1
d
<
Jo... (t, x
(t,z) oo,i, x)
2 j=1
1=1
d
d'
looj (1,2)/
2
d
d'
d
d'
d
(t,
(o,,uu(t, - 2))
2
k=
ot(,
(loo(1,2) (t(,, 0) + (x (o(())))))))(()))))))))))))))))) oo(,,
(0)((,2) - [0
IA3
(ckl (ov))
where (i) follows from Assumption 5 iitmm 2 and the constant bound for (t, 0) in (B.3. Going back to inequality (B77). these bounds implies that
(CG - -2) 20)
< 20-1 (lar + 1)mha (Iua(t, a)) + (t)(t (o(
C(xl + mm (1ua(t, - #o(t, + (c/a) + +av)* loo(t, ott
C(a= +1) (K0,00)) 1)2 + (g)(())(( + 1)20)
< Crg.p (t))(( + (2
20

---

for some C' that doesn't 't depend on 8, where (i) follows from item 3 of Assumption 5
Therefore,
1
- - (s))) ds
01
T
C
!'"
E
x))0(()) 11) ds
10
T
C
T
[8) + 11) (m+ ds
10
T
C
k0 (s)ds
sup (E0.T
E(IX (s) 1) 1)
(B.9)
10
Ola sup 2(m+2)a-1
E eE[o,T]
sup [X (8) (m) (x) +1
AE
C ( (|ag + 1) 1)(m+2)a
S C(22) + 1) (m+
where (i) applies Fubini's theorem due to the positivity of kā and (ii) follows from Theorem K We have shown that this expectation above is finite and independent of 0. Note that, in particular, thi implies that the first expectation in (B.5) is finite as well.
For the second term in the last line of B.7 we first consider for matrix M E R2 and vector E RRd.
a/2
M
Mi
)
a/2
|
1=1
|010
My.d
(nd) |11
M
Apply this inequality, we obtain
{ Voch vx(s ds
E T
d V Vo40,
< 2 E 7
+
ds
CF T /'
d
(1x5(() + 1) 11)
3o no. (8, (8))
10 00.00,..5 (s)) de
+J=1
1/2
(1)
d
n
Σ
N0, (8) i=
10, 0o... (8, (s)) 2a ds
(B.10)
where (i) uses Assumption and the previous matrix norm inequality, and (7i) uses Cauchy-Schward inequality. Let e R be the unit vector with the l'th coordinate equal to 1. Now by Assumption 5
21

---

we have that for fixed > and | = 1........
/ |1ber, (8, (x)) Po.i X (s)) 20++ 20- E T KōeL,0 (1 (s)1) 20 +
E
ds 6 20-5
82ate
By the same argument as in (B.9 this is uniformly bounded Hence
E 1 T
/
1
Mac, (8, (8, (8())) (0, Xx( 2a
(o, (x) ds ds E
lim
ds
T
610
8
A8e, ( (s)) (s, (s(()))
lim E
ds
610
T
8
1
sup
E
Nig00
(0.0
(s) ds
l
1/1
(B.11)
where, again, by the same argument in (B.9). 1/1 is chosen be finite. For the second term the last line of (B. 10), we consider the quantity
- -
Iooissett x) Jo.j (t,
( 0,,, =
oo x)
+ oo,,, x)
2 k=1
8
6
d
(c,,0())
(0, + =l,
(|x) 1)
2
80
(11)
K Sep
dd
dd
(|x| (t)
(t))
(|z) + 0) t
80
((
(|z)
+ 1)
8p
(B.
where (1) follows from Assumption 5 and (ii) applies Jensen's inequality and (iii) recalls the definition in B.3 Note that the reason we define p(p, 1,x)) is because
d
00, 00. (1,
Σ
0o,, (1, (1,2 (0) + go.i (1, 0o.i.
2
s/=1
k=1
lim (22, t,) 810
we see that the same argument B.9 implies the [0, x Q integrability of ((2a o, (*)), uniformly in 5. This then implies that @(2a, i, (z)))) is U.I. for 8 in a neighbourhood of Therefore, we see that
From B.12
['
(s, ds lim E 640
6(2a, 8, 8,
T
1==1
1 T
ko.o(s) 20
(B.13)
sup
0 (1x5 (s) + 1)
T
AE0
< 1/2 < 00
Combining these with B. have establish that
T /' voch (5))) ds S CC ndv,, núa < 00.
(B.14)
In particular, this shows the second expectation in (B.5 is finite well.
22

---

Therefore, in view of B.7
B.9 and B.14 we conclude that
1
( f u (+)) - or ro
U.I. on (1 x [0, T],
F x B[[0, T)), P x LLb)). Hence,
lim E
/
T
1
|cg - - (g(s)) - 6" (ss ds
A-0
T
10]
/
T
lim 0-0
1
| -8) 2o Xi 0" 20 X7()s) ds
[0]
We use the mean value theorem to get that for some C and & = (o,(8, E (0,1).ms- (s, X3 (s)) E (0, 1).
lim
14g - (8) x v0V
8-0
0
d
CC''m lim 8-5
Vo (()
XQ(s))
C lim
Voam (30)) (T(s) - V Vano, (x(1))0, Xf()))
1,/=1
=0
where the last equality follows from the continuity of (0,x) Volle (s, x) and Vedg, (s.
- (s, x) (Assumption [ and 0,0,vo æ) (Assumption 8 and - (s() (Theorem
Therefore, going back to the limit ratio B.6 we have shown that
I' (ce -66-260 -5) vo cols, X5 - 0= [ vo (f(ss)ss =
E
The Jump Part: Similar to the continuous part, we claim that the derivative should be
E
VoLo (o (x)))ss
(B.15)
where VoLy is defined in 3.4 To simplify notation, write
(C) - Ld) (o(s / D1 - D2v(d2)
J
where
Dj == co X3 (s) + xo(s, (z()), - vo(8, Xx + xo (z()).))
D2 Σ (x, (8), (s, (²()), Ovvp Xx (z))
Further, we write == (0 (s), 2) and Xe, == (e. (z((s, 2) when there is no ambiguity in the dependence on s, (s), Then, apply the mean value theorem p vols, X3 (s) + PXe (1 - p)xo), there exists E = Ea (z(,,, z) E (0, s.t.
d
D = Σ
xo. xa.d (co(s, X5(s) + Exo + (1 E)xo),
Therefore,
d
Dj - D2
Σ
xx Xo.d (0.00 (z) Exe + (1 -)xo) Dvvo(s 0,vo(s, (B.16)
23

---

Again, we consider the limit
T (Cd (d) vo( (f(s))d 0² B [ Voldvo(s, (x(ss)))X((s)dd) Xi(s))ds
lim
E
8-+0
lim 0-+0
(C) - (i) vo(s, 0² vo Xg(ss)
10]
lim E 0-40
Jug [z)² D1 D2
Σ Vexo, (0,volt, (,, p(dz)ds
(B.17)
where, as we will show below, the two pre-limit expectations in the first line are finite.
As before, we proceed show that the limit 0 can be exchanged into the triple integration by showing U.I. of
d D1 D2
Σ 8" (ox (s. + Xo) (x(ss)
10)r(2)²
(B.18)
on 1 x [0,
IRg" w.r.t. the probability measure P Leb u(R5T u. We consider
E
1
/ Jeg (2)=)
D1 D2 'e 8" x))) (6(ss + xx) avols, (x(s))) u(dz)ds
T ((Rg"))
T {" ((Rd') Rid 10)07(2)20
D1 D2l ((dz)dd
+ I." ((R())
[
Xō(ss + xo) - (,v0(s, Xo(s))) u(ds)ds
Jag' (2)
=: Ei + E2
(B.19)
We consider the two terms separately. For E,, applying the mean value theorem again (6.16). there exists Th (s, (,), s.t.
d
D1 - D2 =
Σ
xxo xo. + (1 (0x0011,01000 + mexo 7c(1 - E)xo)
1-==1
Therefore,
D1 D2 7(2)2
u(dz)
Egd
4
pla-1)
f
Σ 10,ammoss X²(3) mexo + mm(11 E)xo))
Ixo.i xa. 1Exo.5 + (1 5)
(m m(dz)
Rg
(2)
i-f=1
cc
0,0,vo(s, (s) máxo me(1 5)x01)
u(dz)
Σ
1xo. xo.
2a 1xx..i + (1 1 u((z) 7(2)40
C(11 1/2
24

---

We look at I and 1 separately. By Assumption
9
(5 X3()) nexo + 1/(1 x)xo)
d
H (s, + m55o + n2 x))21
Σ
& (1xg(s) + mEEo 7i(1
=)xo> 1)am
de (1x5 (8) + Ixal + Ixol + 1)
|xm 2m
Xxo
2m
[x]{))] 2m
+
7(2)2m
7(2) 2m
where we recall that 72) = |z| A S 1. Then, we consider, by Assumption 5 for porp23 2
E f
(s.
XB. = sup
uldz) < oo
(B.20)
7(2)
8'E8,
and for all 0, 0' E θ
Ix (8, (x)) (x,
mldz) S (s)
Rgg
7(2)P
So, for p 2
Jeg
Xxo (8, =1)
p(dz) S (1
/
(xxr 0, 2) 7(2)
u(dz)
(2))7
(B.21)
I (s)) + X
As is a finite measure, we have
1 11
1 3
(c (c(IXE(e) 2m
Ix 2m
+ Ixol 2m
7(2)2m0 + 1 :11 mldz)
72)22
< C((X5(s)) (8) 1)
For Iz, we bound
1xo. - Xo,i 20 15xo,5 + (1 - E)xo.s 2a
1 = Σ
u(dz)
722)4a
Rg""
==
[Ex055 + (1 - x) Aa
1xo.s Xu,sI 4ax
1/2
mldz))
7=2)ter ml(z))
7(2)40
Jag'
(1==
1/2
d
1Exo.s + (1- -1=xxx 10
Xo.: Xo.s| M((z)
$
Σ
u(dz)
Jag.
7(2)4a
(22)aa xa.il 2a
j=1
Rdf
1=1
1/2
da
|xo.i dax u(dz) Σ
d
1xo.i
S d2
pl(2))
leg"
(2)daa
leg
7(2)
(1g
|xol + !xol
Ix xol 4a
1/2
m(dz)
u(dz)
7(2)aa
ag
7(2)ac
25

---

By Assumption 5
Ixo Xol 4a
m(dz) S () (s) + 1)da
((2)da
Use this and inequality (B.21) we obtain
d
1xo, Xo.i
1Exos + - -1)xo 2cx
12
p((z)
722)4a
111=1
S (2) + xte, N (x)0() 11 1/2
< C (11 1)
In summary, we have
D1-Di D2 7(2)2
m(dz)= I C(1 1/2
(s) S (sb (m+)))
Therefore, by the same argument as derivation B.9 we conclude that
DDi - -2
sup E sup E
RRd 18)0)(2)2a u(dz)ds
dee
OEE
T JO
< C sup 0Ee
E
da (3)
(x)) (m++2) ds
10
< C sup BEe
14
sup
[s) (m(+
(B.22)
10
00
1/4
Sup
(s)ss)dsss
Aor
(b (m ((mm2))a (|r) + (m + +1)
l:0 10
< C(xx| + 1) (m++aa
where (1) uses Jensen's 's inequality and Theorem Note that, with a 1, this also implies the finiteness of the first expectation in (B.77)
26

---

For the second term in (B.19), we use the same technique as in the derivation for that of the continuous part. First, we consider
d V 20) a xo(s,z, - 0,,0() 1n
Jgg.
mldz)
Rg"
7(2)2a
2a
C
- pldz)
uu
(2))aa
SCI
10,00(5, x xo(0, 2)) - (.)0(s (Vxxor
(x,
Juy (22)20 20
0,0:0088 +x
10 20, (x
2e0 722)20
1=1
SC
1xo x, 2)
Σ
1e (),
Ded 2a2)2a
SC
JuS /
1xo x, x) [H[oo xixo(s,xx Σ
x|]
7(2)20
(1)
1xo(0, 2)
5 C
+ 2x)
+ 1)
72)22a
(161)
Ixo(s, 2,)
2x)
C
(|x) 1)ma
0g Xo, (,,
722)2a d $ C([xl + m)
Rnd
7(2)ma
1xa(,,x,2)1*
10 X0. =)1 ((dz)
7(2)a
7(2)a
Ixols, 2m)
(0, x,, 20 u(dz)
(2)(m++1)a
7(2)a
(x0($,2,2)) uddz) /
1o 20, uldz))
CC(a + ma
Jzz.
(2) 20
7(2)20
Xo(s, xt 2) 12 1)a
u(dz) /
|e (s, I, 2),
Jpg.
ul(zz)
7(2) (m))aam+1aa
7(2)20x
(10)
d
1p. (5, 22 M(dz)
C((z| + (m++jaa
722)2a
Rd
1=1
where (i) applies the mean value theorem to -> (0 pXo) to yield the existence of such & ==
(s,, x. 2), (ii) follows from Assumption 9 and (iii) uses Hölder's inequality ||// & ||
as well as 7(2) < 1, and (iv) follows from B.21 where we have that for 2 2
S/e 1xu(,,I,2)| 7(2)0
m(dz) S C((2) +1y.
ag
27

---

Therefore, by Theorem K and inequality
F2 F. // (BBO)
Voxo, X(()s, (3,00(s, X(((s) + xo) aevo(s, x((s)))
sldz))
Jey' 7(2)2a
1/2
c Σ ΣΣ E { (s))(s)) + 1)(m+1)a (Ls
10, (s), 2) 20x u((z))
ds
{2)
==
1/2
T
10, ()) 2))
Cb
(ja 20m (|z) + 1)(m+1)
E T
(dd2)ds
722)2x
To bound this, as in the continuous part, we check the uniform integrability on Ω x [0. x R w.r.t. the probability measure P Leb - a(BTh when ô is in sufficiently small neighbourhood of of the derivative ratio
1 (2)22a
(B.23)
8
To simplify notation, we again denote Xa. = X0,((8, (z)) 2). To check this, we consider for 2 0
1x80 - Xoo1
2ate
F T
Jeg (22220+e
u(d2)ds
m(Rgd)
5
1
1' 52000 Jdd
[88rq, - Xa,il
2a+e
E.
(dz2)ds
m(Rgg))
T
7(2)
1
E.
(s) + 1)2a 22+e ds
m((R)")) 82a0
T
S glate
(s)ds
E sup (lx) (s) + 1) 200 sE(u,2]
C120+450 + b2a
((|ø) 1)2ate +1
independent of 6. Choose > 0 will show the U.I. of B.23 Therefore, we
1/2
E T u(Reg') Ler Xo.i(s, =)) 7(2)2n
E, s C([x| + (m
u(dz)ds
1=1 1m1
d ΣΣ
f""
[6oe1, x0.l) 2a
= =C([x] + (1) (m+1)a
lim E
610 T m(Re')) leg. 72)2aa
u(d2)ds
mm1
<C([=| + (m++)a
where the last incquality follows from previous derivation with 0. In particular, recalling the definition of Eg in (B.19), this shows that
E ('
Vocfmo(s:x6))) 0
claimed above.
28

---

Therefore, by bounding the two terms in (B.19) we conclude the uniform integrability of B.18) So, going back (B.17). U.I. implies that
!'" (c) (/) vols, (s)
/ T
lim - 0
E
6" E
! ler 1017(z)2
TT
lim E 0-40
Di - D2
0 Voxo, Xx xo) - (x00ss -x0o0)ss)))))))))
u(dz)ds
! Jus 7(2)2
E
lim
D1 D2
8 Voxo. (0,000 + xo) (.,0(s. p(dz)ds
8-0 0
0,
where the last step follows from
1
lim
D1 D2 0T 100Vox0.(s (0ro0s,; + xo) d,vo(s,
0-+0
0
d
Σ
(0volsss6(00 + Xo) 0,0n(s,
lim 0=0
(xo.i(; - ((())) Voxo.(s..
Σ
(0,00(s, (8) + xo) Avall (s))) lim
1 0" (1x Xox),
8-0
0
(8)) 0.
Here, (i) applies the mean value theorem and (ii) use the continuity of 0 - (6, in Assumption 6
The Rewards Part: We first consider the reward rate r. As in the previous proof, we show the U.I.
of
1
(00)=
E T
lpo(s, X²())) puls, (x)) ds
|0
and the finiteness of
( ds
0
T
for some > 1. By Assumption 7 Nitem 1 and the same derivation as in B.9
C E
( n (())(()))))())) + 1)mads ds
(0) <
10 T
C
ro,os)ds
sup (E0.)E(D.T]
E([X{(s) 1)ma
10
< C(1l + 1)ma
uniformly in 8. Moreover,
1 E
{ T T
/ (00,
Téeu (s, Xg) (x(
E
lim
ds
T
$10
8 Tōem (8, X (s))
T
(o, (s))s)
lim 810
ds
T
1
[' T
< sup
E
10.0 (x))s))))
0
T
< 00.
29

---

These results and the continuity of (0 + Vor(s, and 0 - (s) implies that
no (8(s 0T F. / Vopo(s;X(())00)
lim
F
po(s,
000
[
E
lim
po(ss
Xx 8²
ds
[8]
0.
For the terminal reward term, the same proof with the integral removed and replaced by T will yield the desired conclusion.
C Proof of Proposition A.1
We note that the statement for X5 and its first derivative holds from directly applying Kunita [|] Theorem 3.3.2] and the a.s. version of Kolmogorov's continuity criterion in Corollary I of Protter 19 Theorem 73].
To show the statement for the second derivative, apply the proof of Kunita From display (3.43), we look at the random drift:
Theorem 3.4.2].
M
Ho,b) = Σ
m=1
seen as function of r E [0, H E RRdxdxd, and show that it satisfies the conditions for Kunita 11 Theorem 3.3.2] with À = 2;; i.e. the conditions in Assumption TO
We note that as μl and 0m æ)) satisfying item Assumption 10 for any L, m I.....de
Mo x + dem) - Mo x))
sup re(o,7]
(r. = sup
lim re rE(0,7] 610
8
lim |oo(r, + dem) mo(r,
sup
rE[0,7] 810
5
is bounded. Same for i. impo
First, at Ha,b 0.
sup rE(o,T]).ERRd
(r, (r, S sup rETo,T].EEB
El
< co.
m,t=1
Second, M (r, Ha,)) is clearly uniformly Lipschitz in Habb 0pp bounded.
Third, using the boundedness of 0m and Oppo. we have
S (1 (((s,)))((
Xg((s, H),)
30

---

and
mm
on On. (s. ( (1)0)) (s,
m,1=1
+ 10m (r, X (s, (s) X 04
- 0u (,. (s.r)
Σ - (s,, (0,7) C C00xx O. -0, 10x00
m,1=1
+ - - - 0, 10.0,m
Therefore, defining (r, (r, H) to be the sum of the two, we see that
F / K (r, (r, dr C| y/ C| y/p < Cla v)P((Ha, s 1)P.
Here the first inequality follows from the first derivative satisfying the Proposition A.1 which follows from direct application of Kunita [T] Theorem 3.3.2].
Similar results can be established for the volatility and the jump coefficients. Therefore, we conclude the proof by applying Kunita [I]] Theorem 3.3.2] the derivative and the second derivatives.
D Proof of Theorem 2
Our proof of Theorem 2 hinges on the ability to exchange the derivative with the expectation and time integral. To achieve this, first, similar techniques in the proof of Theorem to prove the following lemma.
Lemma 2. Under the assumptions of Theorem 2 for h(t, po(t, pt(t,)))))))))))) and go( we have
(D.1)
and
0,, Oz,Oz, Eh(s, (t, E0, Ms, Ea,Xx (t. s) H[M)(s
Ms, (t,
(t(,,s)) (t,
(D.2)
Moreover, there exists a constant C independent of s.l.
E)O,, h(s, X5 s)) & C(x2| + 1)"" and EI h(s, Ns, s)) s C([a] + 1)
Lemma directly implies that the derivatives of the expected terminal rewards in 2.6 satisfy
Fgg (t, X (t, 7)) E (t)(t, T), H2 [Ego (t. (t, T)) = (t, To Hogo (V H [t. T))]
(D.3)
By the same argument, to prove Theorem 2' it suffices to show that for the cumulative reward parts in 2.6p. the time integral and space derivatives be interchanged. First, by Lemmal 2 we see that
f Elor,po(s, (t,
Xx.pposssxtt... 8)) ds < co, and [ Fl0,,0n,Pol (t, s)) < co.
31

---

So, by Fubini's theorem and Lemma 2
E L Oz,polo, (1 (1,8)
[ Eoo,,o(s, (1(()))dd
= J." Ep, X (t, s))ds
On. [ Epo(s, X (t, s))ds
Dz, E S.² Xo (1, s))ds
where (i) follows from dominated convergence that for y in neighbourhood of x,
Oy Epo(s, (1, 8))1 & Pos, s))1 3 C([=] + +e+1))
independent of Similarly,
=f. 02, Xp s))ds = dz, E S( st)
E
This and (D3) implies (2.6). completing the proof.
D.1
Proof of Lemma
First Space Derivatives: We first show equality (.I) Consider
1
0, E.(s X
(t s)) = lim E Mss Xo, (t) h(s, ((s,
(D.4)
8-0 8
We exchange the limit and the expectation by considering
E8-a a h(s, X² tgtton s)) hss (t),s)))
Es
Vh(s, EXO Exx (t, + (1- - (t, 8))
(xotdo - -xt(t,
1/2
Xo xattes
(1, ()) 2nx
E
EIVh(s,EX0
+1 (1- (1, 2))
5
where the mean value theorem implies the existence of such r.v. & E [o, 1]. For the first term, Proposition A. iimpliss that
x + bo; (t,
2a (t,
2n
For the second term, by Assumption
EX,
(t, + (1 x) (t, s)) 20x
c E( + on; (1,5) + |X (1, s)1 + 2am
2a
0
< c. E(Xx (t, X6 s) + 2 2| (t, s) 2am
(D.5)
< C(xx| + 1) 20m + C12am 2com
S C(xx| + 1)a0 2am
where the last inequality considers [8] and can be chosen so that it doesn't depend on 8, 8, and 1. Therefore, the limit in the r.h.s. of DD4 can be interchanged with the expectation and we have that
On Eh(s, X (t, 8)) = E0,, (t,
(V (t,,, To, (t,
32

---

Also, the previous derivation with a = 1 and taking the limit 5 - 0 implies that
h h(s h( < C(1a] + 1)m
where C' doesn't depend on and t.
Second Space Derivatives: Then, show equality (.2) Previous proof implies that
02,02, Eh(s, 8, = 0z, EVh(s, ()), ((x),))))xxx(((3))))
Hence we employ the same strategy to exchange the limit and expectations for the following expres- sion
lim
E
Vh(s, ( (t. T..00 (0 (t, (t(,ss a,x,
6000
8
lim E 0:000x0 (1, 8) (Vh(s, x2t66, (1, - Vh(s, Xi (t,
(D.6)
6-40
+ lim Vh X 5 (t, (0.X0 2 (t, 0,x 8))
3
We show U.I. for the two terms in D.6 separately. For the first term, consider
j (1,
(Vh(s, Xo
x2 (1,
(1(,, (1)
20 1/2
E 0
20
X0
(t. 8)) x
Vh(s, (t, s))
82a
By Proposition consider
A.I the first expectation is bounded uniformly in s and t. For the second term,
Vh(s, Xo Xxtbes (t, X 2a
22
8, X x=+t6es (1, - 0ch(s, X (1, ()) 2
82
(1) 1
1x (xot8es(11 (1,8) - -X 2
Ti,h(s,&iX 805 (1,8) + (1-
82a
X xatóes (t, Xt 20
Hihl(s, +66 (t, (1 ( (t. s)) 20
82
(4))
X xzt (t,s) - Xg s)
20
(1xx (t. s) X (t, s) + X tt. 8) + 2am
where (i) follows from the mean value theorem with n.v. E [0, 1). and ( applies Assumption 4 Therefore, we have that
katto (1, - Vh(s, (1, 20 2a(m+1)
F 820
Vh(s,
Xo
CF
1
x
80, (1,8) (1,(, 8)
J20
x22 ((.)) (t, da 1/2
+C
E ((( s)l 1) dam E fla
s cc <c ( + C(1 |x|) 20(m+1)
20m
where the last inequality follows from Proposition A.I showing U.I. for the first term in (.6).
This is uniformly bounded in -
33

---

For the second term in D.6. we consider
E
Vh Xo (t,ss (t,s)) (0,
Xo
(t, 0,x (t,
E Vh(s, Xo xottee (t,s)) 2a
10:x0 (1, - -1xx5(1,8)) 20 1/2
FE
82a
1/2
ch [((xX (t, +1) 1)2am)
I C(22am (| + deşl + 1)2am 1) 1/2 which is also uniformly bounded in as + O. Therefore, exchanging the limits in D.6), obtain
Or, Or Eh(8, Xo(,, 8, x)) = FO, (1, () (s X (1, 8))0, 8) + Vh(s, (g(,, aa, *))
Moreover, by setting a = 1 and taking the limit as 8 + in the preceding derivations, we see that E 2,, 0z, xo,,, x))] v C(xl 1)th where the constant C' is uniform in and t. Proof of Theorem
E
From A.
we see that
E (o = E = (o
Ef T
Moreover, since r independent of F,
E f VoLoVo(4, = /'" VoV 1) 7
Therefore, by Theorem ED(x) = (0)) For the variance, we consider
1 Vol -1
= TE
VOLoV (00
V 2 ((t, ()) (X00)
J0
('
||1 (BV (1(, 1/2
<
1/2
dtt + I' Ez((t)(((000))
0
T J
E|Voho
1/2
1 T
ddt 1
T l' E) (t
+
T T
V
By B.Il and (B.13) with 2,
['
E E
< sup
(g)os)dds sup E(IX((s) + +1) < C(22 + +1) sE(o.T]
T J
T 0
10
34

---

and similarly
T /' E Vooll S C(|2] + 1)a.
1
By definition and Proposition A. have that
S CE J.
+ " + (1 [X](()))))
s 2CT sup
rE(O,T]
< C([2l + 1)am.
Similarly, (t, (0, t)) & C((2} + 1) These calculations imply that E Vo (0,
(0,r)) +))1 < C(|r) + 1)2444.
For the reward rate and terminal reward terms, we recall Assumption 10 with the additional Assump- tion that a > Note that since a > 2, for
(to(t, x)( = (po(t, - xt)
S ()) (2000))/0000)0 (|x]| 1)
So, have that
2
(
F
V
Napol
2d
S
lim
E
Po0 pol 2dl
00+0
J
|0|
T
sup
E
PP
2ol2
10
0 J
sup
102 (
(s)
2/a dt
sup (0, + 1)3m
AE
rE[U,27
2/a
10p2 (s))d
C(xx|+1) + 1) 2m
sup
6
S
((x]
+ 2m
where (i) uses a > so that the integrand U.I. in 0 E (see (B.11 for a similar proof), and (10) uses Jensen's inequality with 2/a < 1. The same holds for the terminal reward term, with Ng = 1 0¹1" integrable.
Therefore, we conclude that Var()D(z))) < D(x) < C[x[|+))mm+ where C can be dependent on other parameters but not x.
F
Supplementary Materials for Section
F1 Calculating the Estimators
The Generator Gradient Estimator: We compute
-a (t = E J. dc, - (x,g((
F. (" X0 Vun (1,
+ L." X5(1,)) (Q 20
(1, xx(,7)
(Qr + (001)))((1++11
35

---

Here (t. s) is column vector. So, replacing it by the Jacobian will yield row vector. Following the definition of Z in (2.6 we define
Z(t,z))= { uols Xx R)) R (
(F.1)
+ f (1, 8) (Q + V + (1,
(1,
( Q+)
Here, the derivative process VX3 satisfies the following ODE with random coefficients:
(1, ei + I (A+ BV (x(4))))
or in matrix form:
(x)(4,))11 =1 1+ J. (A BVus(r, VX)
Therefore, in this setting, our generator gradient estimator in 2.9 is
D((x)= To,, us (T, (z)))) B + (T)) )oo us (o( (F2)
where Z is given by F.II As explained in (2.9) we also randomize the integral corresponding the gradient of the reward rate VePo-
The Pathwise Differentiation Estimator: From randomizes the time integral
construct the following IPA estimator that
Dx(x) = Tuo(T, X3 + - (0,, + TT, X (T) (R+ R 10, 00, (o ( Xg(T)) + (g) (Qr
0,,,
Here, the pathwise derivatives 0n, (1) is the solution the following ODE with random coefficient:
0, / (A BVa X8(5)) + Bθ,, us(s, X3 (s))sssss
(F3)
F22 Numerical Experimentation Details
We conducted the computation time and variance comparison for both estimators using PyTorch. The computation time data was generated on system equipped with a PCIE version of Nvidia Tesla V100 GPU, featuring 32GB of VRAM. Additionally, the system includes 2-core CPU and 16GB of RAM, which are used to format and store data. The primary computational tasks are handled by GPU.
The data for Table iis produced follows. For each n, we produce 400 i.i.d. GG and PD estimators (D6)) (zo), (D)) (zu) IR" l...., 400 Let
2
400
400
1
Σ
(ro) (
1
Σ
D
((
(xo)
20
400
j=1
j=1
400
400
ŌPD
1
D (ao) (xr)
1
Σ
(1)
(xx)
20
400
j=1
j=1
The "Avg SE of GG" and "Avg SE of PD" entries record
ooa,s and
deG.
(F.4)
respectively. The "Avg SE ratios" compute
ōGG,
(F.5)
7
ŌPD,
36

---

1.02e+02
1.00e+000 +3
10
Pothwise Differentiaion
Generator Gradient
Density
10-2
10-3
10"4
n 5.50e+03
n 2.10ee 04
10-1
10-2
Density
10-3
10
10-5
3.24e+005
1.29++06
10-2.
Density
10
10-6.
1.15e+07
10-2
Density
10-1
10-6.
100
200
300
0
100
200
300
Standard
Standard Error
Figure 2: Histograms comparison of the distribution formed by the standard errors of coordinates of the estimators. These plots use the same data as that produces Table
The numerical values used for the matrices, initial conditions, and network initializations for the SDE models can be found in the supplied code.
We further analyze variance by plotting histograms of the distribution formed by the standard errors of the coordinates of the estimators, shown Figure 2 The standard error distribution of the pathwise differentiation method exhibits a heavier tail compared to our proposed generator gradient estimator. This aligns with the superior variance performance our estimator demonstrated Table 2 also provides insights into the confidence intervals in Figure Tb which are barely visible due to high confidence levels. particular, the generator gradient estimator has tighter confidence
2 Figure
intervals Figure
lb
37

---

G Experiments on SDEs with Non-Differentiable Parameters
G.1
CIR Model
In this section, the Cox- Dox-Ingersoll-Rowe Ross (CIR) diffusion example to test the validity of the proposed generator gradient estimator when the differentiability assumptions of the coefficients violated. Specifically, consider the one-dimensional process:
(t, = = x J, (t,rr)) J. (ttttr)
(G.1)
Note that the volatility x(t vz is not differentiable at though it is C for > 0. A unique strong solution to G..I always exists. Moreover, if 2 1/2. then (t) for all E [0, 2] almost surely.
for t E [0,2]. 2]. where 2,0
We consider the following value function:
Cy (t, x) xt) = E [ X (t, s)ds
(G.2)
We aim to estimate the gradient Vovo x) evaluated at x 0.1 for multiple values of 6.
Since the pathwise differentiation estimator also suffers from non-differentiabiliy issues, validate the generator gradient (GG) estimator by comparing with the finite difference (FD) estimator. Specifically, the FD estimator computes
1 h
(0) = h. 100 (0, - 10- (0,
(G.3)
using Monte Carlo simulation of the SDE G.G and the value function. In this context, the FD estimator should consistently estimate the gradient evaluated at any > The GG estimator is produced from (.9). We use the Fuler scheme to simulate the SDEs and the derivative processes. To avoid numerical issues when the Euler discretization of the CIR process crosses 0, we take the absolute value of the discretized process at each time step.
Table 3: Statistics for 10 sample averaged GG and FD estimator. For the FD estimator, we choose h 0.05 in (G.). resulting in a bias of Oh??).
Value of 0
0.55
GG + 95% CI
1.135 + 0.0011
1.135 + 0.0012
1.134 = 0.0019
FD + 95%
1.095 + 0.064
1.097 + 0.046
1.116 + 0.026
Value of 0
0.45
0.2
0.1
GG 95% CI
1.134
0.0023
1.179 + 0.112
1.492 - 9.759
FD + 95% CI
1.112
0.024
1.020 + 0.018
0.895 = 0.015
Table Blsummarizs the estimated value and confidence interval for both the GG and FD estimators. We note that bias of O(h?) is present in the FD case. When 0 > 1/2, observe that even though Assumptions Tand 3 are violated, the GG estimator produces results consistent with the FD estimator. This suggests the validity of the GG estimator even when Assumptions and don't t hold. This consistency occurs because, this case, the derivative processes are still well-defined up to the first time Xı (t) hits O, which does not happen when 0 1/2.
However, when < 1/2 (cases highlighted in blue in Table 3. the sample paths of the SDE G.I can reach O. Although the statistics in Table 3 appear consistent, we observe a significant increase in the variance of the GG estimator as 0 decreases. This increase may indicate that the GG estimator not consistently estimating the gradient in these cases.
38

---

G.2 SDE with ReLU Drift
In this section, we use the following SDE with ReLU drift as an example to test the validity of the proposed generator gradient estimator:
(t. - + J. (R), + 1)dr + J dB(r)
(G.4)
for t, E [0, 2), where > and we choose a -0.1. Note that the +1 in the drift makes it always positive. So, starting from -0.1, the process should cross 0 (where the drift is non-differentiable) before time 2 with high probability.
With defined in G.2 aim to estimate the gradient Vavg (0, evaluated at 0.1 for multiple values of using the GG and FD (defined (G.3) estimators.
Table 4: Statistics for 10 sample averaged GG and FD estimators. For the FD estimator, we choose 0.05 G.3). resulting in a bias of (hh)).
Value of 0
2
0.5
GG + 95% CI
14.91 + 0.031
4.087 = 0.008
2.300 + 0.005
FD + 95% CI
14.78 + 0.570
4.131 = 0.192 2.394 + 0.127
Table summarizes the estimated values and confidence intervals for both the GG and FD estimators. Note that a bias of O(h²) is present in the FD case. Despite violations of Assumptions | and 3 the GG estimator still produces consistent results compared the FD estimator. We note that this context, it should be possible to establish the existence and integrability of the derivative processes for any value of 6.
39

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The context and contributions of this paper are clearly and accurately stated in the abstract and the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We acknowledge the specific parts of the assumptions that could potentially be stronger than what necessary. See, for example, the remarks and discussion following
Assumption I and 2
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
40

---

Answer: [Yes]
Justification: All theorems, propositions, and lemmas are either proved within the paper cited from other works. The assumptions are clearly labeled and discussed. We also provided intuitive justification for our theoretical results in Section 2.
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The code attached to the submission, if run properly with system configurations similar that indicated in the appendix, will produce very similar qualitative results as that presented in the paper. However, as the neural networks are randomly initialized, the quantitative outcome might differ.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibilitty In the case of closed-source models, it may be that access to the model is limited in
41

---

reproducible, they some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
The code is submitted with the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
wwblic/guides/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips htt.cccpppbblccc httpppullllciisieeepppppppppe guides/ (ooeeeumisssiono ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:[Yor Yes]
The parameters of the control system in Section are presented in the code. There is no hyperparameter that needs fine-tuning.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: Although we do report data involving error bars (see lb these are barely visible, and have no qualitative significance to the numerical results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
42

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
This is fully specified in Appendix F.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https:://neut http://uuurips.cc/pb/b//p hccppublic/Bchcccccccee....
Answer:
[Yes]
We conform to the NeurIPS Code of Ethics. Anonymity is preserved this submission.
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work makes methodological contributions to the SDE gradient estimation problem. There is direct social impact associated with this work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
43

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer:[NA]
Justification:
There is no high-risk data this paper.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA].
Justification:
The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
44

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
45

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
46


Reviews:
[{"review_id": "0rushKdQqH", "reviewer": "Reviewer_rGnk", "summary": "This paper formulates an efficient, unbiarsed, and finite variance gradient estimator for an objective function that looks like the stochastic optimal control cost function (it is ubiquitous across various applications). The problem concerns overparametrized SDEs (the paramete dimension n r is of a much higher dimension than the state space d).  The algorithm runs at a complexity invariant to the large n. The authors use the Feynman-Kac PDE for v_{\\theta} and transform it to an equivalent PDE to solve. They then employ a pathwise differentiation estimator to estimate the 2nd order and 1st order derivatives in the PDE while exploiting certain properties of the Hessian.", "questions": "1. Are there any error bounds from using the probabilistic representations for the spatial derivatives (see #4 in Weaknesses)\n\nHappy to revise score if necessary.", "limitations": "Yes they have mentioned some, rest may be in the above Weakness section unless there are some misunderstandings.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The applications of gradient estimators and the motivation behind this problem is well stated and supported. \n2. The methodology is clear, especially when finding probablistic representations of the gradient estimator.", "weaknesses": "1. The authors discuss the limitations in the checklist but it would have been more clear to have a limitations section clearly outlined as claimed in the checklist.\n2. The assumptions may be restrictive as mentioned. \n3. The work could be improved if there was a numerical simulation. The example of LQR is an illustrative example and it may be of didactic use and potentially relatively easy to implement a scenario when the parametrization is of a much higher dimension. \n4. The spatial derivatives \\partial v_0 are still difficult to compute, but still better than \\partial v_{\\theta}. What are some error bounds you can get by using the later derived probabilistic representations that can use Monte Carlo", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and questions. We organize our responses as follows\n\n1. **Limitations induced by the assumptions on the smoothness:** Thanks for the suggestion of including a separate discussion of limitations to avoid misunderstanding. In the \"Author Rebuttal\" section, we address concerns regarding the clear presentation of the weaknesses induced by requiring additional smoothness in the SDE parameters, as well as the proof techniques we can adopt to circumvent the theoretical challenges when smoothness in the SDE parameters is lacking. This is also confirmed by our additional empirical demonstration using the CIR process and SDE with ReLU drift. These new experimental results suggest that our generator gradient estimator remains consistent with finite variance even when the SDE parameters are not smooth, indicating a wide applicability of our proposed methodology.\n\n2. **Error bounds:** As we have explained in the \"Author Rebuttal\" section, the typical worst-case upper bounds produced by analyzing the numerical SDE schemes, applied to the derivative processes, while reassuring the validity of the proposed estimator, cannot explicitly demonstrate the variance and bias dependence on the dimensions. Moreover, we believe that the error's dependence on these dimensions is instance-specific, depending on the SDE and reward functions. These dependencies are technically challenging to explicitly reflect in an error bound.\n\n3. **High-dimensional parameterization:** For the LQG example, we have tested it in an extremely overparameterized setting where $n$ the dimension of $\\theta$ is more than $10^7$. In this setting, our estimator outperforms the pathwise differentiation estimator in computation time as well as variance. We can also provide a demonstration of running gradient descent using our estimator, plotting the terminal performance of the control policy. If this could resolve your concern, please let us know. \n\nWe believe that these additional discussions and clarifications enhance the quality and clarity of the paper. Additionally, we are confident that our methodology will attract interest from various research communities, and sharing our work at NeurIPS will help achieve this goal. If our response addresses your concerns, we kindly request that you consider raising your score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2ZPMmc8Oot", "reviewer": "Reviewer_Eiq9", "summary": "The paper proposes a novel algorithm for the gradient of machine learning objectives that are based on SDE paths, w.r.t. parameters. The method is scalable for big parameters, as the computational complexity of the estimation is not related to the number of parameters. Empirical results show the scalability advantage over classical pathwise differentiation.", "questions": "See weakness. Also, applying the Feynman-Kac twice requires the objective to be very smooth, can you provide some insights on the applications where this holds true?", "limitations": "As written in questions, the smoothness requirements induced by Feynman Kac etc. will limit the use cases.\nAs written in weaknesses, it is still unclear the convergence w.r.t. high number of parameters.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "PDE constrained optimization has remained a very difficult question due to the high cost of simulating PDEs, let alone taking derivatives. Therefore, despite its many good theoretical promises, it has not been deployed. This paper attacks the question by reducing the computational cost with a monte-carlo estimation of the derivative, largely advances the promise of PDE constrained optimization. \nThe writing is also very good, and the intuitions behind the methods are well illustrated.", "weaknesses": "1. Despite the fact that the computational complexity is irrelevant to the number of parameters, I suspect that as the number grows, the method would require more samples for an accurate monte-carlo estimate of the gradients.\nIs it possible to either:\n1. Provide a theoretical analysis on the convergence rate of the estimation w.r.t. the number of parameters.\n2. Provide an empirical analysis on the convergence rate of the estimation w.r.t. the number of parameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and questions. We hope that the following response can address some of your concerns:\n\n1. **Limitations induced by the assumptions on the smoothness:** \n   In the \"Author Rebuttal\" section, we address concerns regarding the clear presentation of the limitations due to requiring additional smoothness in the SDE parameters, as well as the proof techniques we can adopt to circumvent the theoretical challenges when smoothness in the SDE parameters is lacking. This is also confirmed by our additional empirical demonstration using the CIR process and SDE with ReLU drift. These new experimental results suggest that our generator gradient estimator remains consistent with finite variance even when the SDE parameters are not smooth, indicating a wide applicability of our proposed methodology.\n\n   Regarding the application of the Feynman-Kac formula, we use a probabilistic proof based on martingale methods. This approach requires applying the Feynman-Kac formula once to obtain the martingales in Lemma 1, taking the limit of the difference quotient, and proving uniform integrability to get the $\\theta$ gradient. The introduction that uses PDE techniques is only for intuitive understanding. We note that probabilistic tools can sometimes yield weaker smoothness requirements than functional analytic methods.\n\n    That said, our approach does require that the value function is a classical solution (with possible relaxation that the second derivative is continuous almost everywhere) to apply Itô's lemma and the Feynman-Kac formula (see Assumption 2). Nevertheless, Assumption 2 can be established using a stochastic flow argument as in [Kunita 2019, Chapter 4] (cited in our paper) if the rewards are sufficiently smooth and the derivative processes are integrable. It can also be established using PDE arguments (see [Evans 2022], also cited in our paper). Due to space limitations, we did not include a full discussion of how this is done and instead presented it as an assumption. This is discussed in lines 196-201 after Assumption 2.\n\n\n2. **Convergence properties as the number of parameters increases:** The variance behavior of our algorithm, when the number of parameters becomes very large, could be a concern. Therefore, we included Figure 2 in Appendix F.2, showing histograms of the standard errors of each of the gradient coordinates under a wide range of $n \\in [10^2, 10^8]$. We see that the tail of the standard errors grows from about 25 when $n \\approx 100$ to about 200 when $n \\approx 10^7$. We believe that this is not a prohibitive increase in variance, considering that the number of parameters is many orders of magnitude larger. This, coupled with the sketched error analysis in the \"Author Rebuttal\" section, suggests that the increase in variance is manageable. Additionally, the generator gradient estimator has a lighter tail compared to the pathwise differentiation estimator, suggesting a uniformly better variance performance in this setting. \n\n    However, we should note that the standard error distribution depends on the $\\theta$ where we evaluate the neural network. Here, we use the default initialization provided by PyTorch.\n\n    Even though the generator gradient estimator doesn't need to simulate an SDE of dimension linear in $n$, we still need to evaluate the neural network and compute $\\nabla_\\theta \\mu_\\theta$ and $\\nabla_\\theta \\sigma_\\theta$, which could be costly if $n$ is large. However, upon investigation, we find that our proposed estimator $D(x)$ evaluates $\\nabla_\\theta \\mu_\\theta$ and $\\nabla_\\theta \\sigma_\\theta$ only once. In contrast, using pathwise differentiation requires evaluating $\\nabla_\\theta \\mu_\\theta$ and $\\nabla_\\theta \\sigma_\\theta$ at each time step to simulate $\\nabla_\\theta X_\\theta^x(t)$. We believe this is why our estimator has a very stable computation time even when $n \\approx 10^7$.\n\n3. **Error bounds:** As we have explained in the \"Author Rebuttal\" section, the typical worst-case upper bounds produced by analyzing the numerical SDE schemes, applied to the derivative processes, while reassuring the validity of the proposed estimator, cannot explicitly demonstrate the variance and bias dependence on the dimensions. Moreover, we believe that the error's dependence on these dimensions is instance-specific, depending on the SDE and reward functions. These dependencies are technically challenging to explicitly reflect in an error bound. We hope that the previous empirical validation of the variance behavior can alleviate your concern. \n\nWe believe that these additional discussions and clarifications enhance the quality and clarity of the paper. Additionally, we are confident that our methodology will attract interest from various research communities, and sharing our work at NeurIPS will help achieve this goal. If our response addresses your concerns, we kindly request that you consider raising your score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "amZFRONoyQ", "reviewer": "Reviewer_5TVX", "summary": "The article studies classical stochastic control in the continuous d-dimensional SDE setting (with and without jumps). The authors suggest to compute the gradients of the value function (expected total reward in the non-discounted finite time regime), which can be applied (in a model-based setting) to policy gradient type algorithms popular in RL. While in a straight-forward fashion computing the derivatives of the expectations would lead to the simulation of n (dimension of approximation space for the control) expectations that require the simulation of one SDE each, the authors propose a clever interchange of derivatives that reduces the number of SDEs to a polynomial in d. Very careful proofs are given, the findings are applied to simple linear SDE example with quadratic cost function.", "questions": "1. You will typically have to solve the three SDES using numerical schemes. Can you provide additional assumptions on the model (the coefficients) that, combined with SDE numeric results (let's say in the Brownian case only), provides a result for the estimation error of the gradient? \n\n2. The additional derivatives on the model coefficients make me doubt the SDE scheme (2.7) works well if the coefficients are not so nice. Many control problems in applications are Heston type models, which are already problematic for numerics with the square-root variance coefficient. I know they do not satisfy the assumptions imposed, did you still try your simulation scheme?\n\n3. I do not understand why you included the jumps without an application in mind. This is normal for a maths journal, but a bit strange in ML/AI. Could you please provide an example?\n\n4. A few minor points:\n(i) The generator of a Markov process is the action plus the domain. Could you please be a bit more careful when you rework the article?\n(ii) You might want to mention that uniformly bounded derivatives imply linear growth which is the standard property required for global existence.\n(iii) I am probably missing where you discuss properties on the model that imply Assumption 2. If not I find assumptions on the derivatives of the mean rewards disturbing that I do not know how to check.\n(iv) Typo in Assumption 1, 2.\n(v) At least for the larger expectations it would help to be more consistent with the use of brackets.\n(vi) Could you please put references to the quadratic loss examples? Since this is the only application it would be good to see where and how it is used.\n(vii) the use of colons is not very helpful, for instance in (2.6) it is a bit complicated to understand what object is defined (I guess you want to define Z).\n(viii) The chain rule is missing in (D.3)", "limitations": "The authors did not discuss the limitations or the practical problems of the results.", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "The article introduces a creative switching trick (switching order of derivatives and derivatives with expectations) that allows to reduce the simulation dimension compared to a straight-forward estimator.\n\nThe rest of the paper consists of a rigorous verification of interchanging limits (derivatives) and expectations. The proofs are careful (mainly based on differentiability properties of SDE flows), I spotted only few typos.", "weaknesses": "I do not agree with the comparison to RL (=sample based stochastic control) in Section 1.1. The entire point of RL is to provide model-free estimators, in particular, the policy gradient theorem cancels out the model transitions. Here, the situation is different. The model ($\\mu$ and $\\sigma$) is used to write down the SDEs (2.7) required to simulate the estimator. This is not a problem but shifts the article more towards the stochastic control community.\n\nI cannot see a clear interest in the ML/AI community in the questions raised, the control setting of SDEs (jump SDEs) is natural in the probability/math finance community but not at all in ML/AI. Even if I try I struggle to come up with examples relevant for the ML/AI community (the authors neither).\n\nThe article is written for a purely mathematical audience with research level skills in stochastic process theory, the article could appear almost unchanged in journals such as SPA or SIAM Journal on Control and Optimzation. I think NeurIPS is not the right place for publication. I am willing to increase my scores if a more relevant example than the quadratic problem could be provided that shows value of the switching trick to the ML/AI community.\n\nThe appendix is chaotic. It is tricky to follow the proofs because parts of the proofs are deferred to other sections of the appendix. Given that the proof only justifies interchange of limits and expectations it looks much harder than it is.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments. We hope that the following response will help to address some of the concerns. \n\n1. **Applicability and interest to the ML community:** Recently, there has been significant interest in approximating continuous-time optimal control using neural networks and gradient methods, with applications in finance [1], scheduling of queuing systems [2], and stabilizing stochastic systems [3]. Our method offers a new scalable approach to approximate the optimal policies of these complex stochastic control problems. Additionally, our methodology can be applied to many ML-in-science contexts, including PDE-constrained optimization [4] and neural SDEs [5,6,7].\n\n\n2. **Stochastic control and RL:** We note that our literature review in Section 1.1 aims to acknowledge and compare previous research on Monte Carlo gradient estimation and its applications. We consider both policy gradient methods in RL and gradient methods in stochastic control to be important examples. However, we do not intend to claim that they are equivalent. We will clarify this by improving the writing in this section.\n\n\n3. **Applications of jump diffusion models:** Regarding the reviewer's concern about the applicability of diffusions with jumps, we note that many models in the finance literature require the inclusion of jumps to accurately capture market behavior [8] (cited in the paper). Therefore, ML models for financial applications would benefit from the versatility of including jumps. Moreover, continuous-time Markov chain (CTMC) models are widely used in chemistry, biology, and physics to model the stochastic behavior of natural systems (see [7] and references therein, also cited in the paper). CTMCs are also a valuable tool for analyzing the behavior of discrete event systems. The jump-diffusion model considered in this paper captures a large class of CTMCs, underscoring the importance of including jump behavior in the SDEs we study.\n\n\n4. **Assumptions on the smoothness of SDE coefficients:** We address the concerns regarding numerical error analysis and the applicability of our methodology when the SDE coefficients are not smooth in the \"Author Rebuttal\" section. In particular, we have tested our estimator for the CIR process with a square-root volatility, yielding results that are consistent with our expectations. We believe that the generator gradient estimator remains unbiased with finite variance for a wide range of non-smooth SDE models, and we support this claim with numerical experiments. We hope these additions will alleviate some of the concerns.\n\n\n5. **Typos and the organization of the Appendices:**  We thank the reviewer for their careful review and for pointing out typos and presentation issues within this paper. We will address these issues in the revision. Regarding the organization of the appendices, we aim to first present an overview of the important supporting results (presented as lemmas and propositions) and discuss why proving these results will imply the main theorems. We defer the detailed proofs of these supporting results to the appendices for interested readers. These proofs are typically rigorous checks for the uniform integrability of the pre-limit expectations and are lengthy due to the presence of many terms. We note that in order to get the correct polynomial growth order of the variance in Theorem 3, a careful analysis of the growth power of the parameters and rewards is required. \n\n6. **Establishing Assumption 2:** Regarding the difficulty of checking Assumption 2, it can be established using a stochastic flow argument as in [Kunita 2019, Chapter 4] (cited in our paper) if the rewards are sufficiently smooth and the derivative processes are integrable. This can also be established using PDE arguments (see [Evans 2022], also cited in our paper). However, due to space limitations, we didn't include a full discussion of how this is done and instead presented it as an assumption. This is discussed in lines 196-201 after Assumption 2. \n\nFinally, we have decided to submit this manuscript for publication in the proceedings of NeurIPS to share our research output with a broader community. We believe that our methodology will find exciting applications beyond our domain of expertise. This aligns with the spirit of NeurIPS.\n\nWe believe that these additional discussions and clarifications enhance the quality and clarity of the paper. If our response addresses your concerns, we kindly request that you consider raising your score. \n\n[1] Fan, Lei, and Justin Sirignano. Machine Learning Methods for Pricing Financial Derivatives. arXiv preprint arXiv:2406.00459 (2024).\n\n[2] Ata, Baris, and Ebru Kasikaralar. Dynamic Scheduling of a Multiclass Queue in the Halfin-Whitt Regime: A Computational Approach for High-Dimensional Problems. Available at SSRN 4649551 (2023).\n\n[3] Zhang, Jingdong, Qunxi Zhu, and Wei Lin. Neural stochastic control. Advances in neural information processing systems 35 (2022): 9098-9110.\n\n[4] Sirignano, Justin, Jonathan MacArt, and Konstantinos Spiliopoulos. PDE-constrained models with neural network terms: Optimization and global convergence. Journal of Computational Physics 481 (2023): 112016.\n\n[5] Tzen, Belinda, and Maxim Raginsky. Neural stochastic differential equations: Deep latent gaussian models in the diffusion limit. arXiv preprint arXiv:1905.09883 (2019).\n\n[6] Kidger, Patrick. On neural differential equations. arXiv preprint arXiv:2202.02435 (2022).\n\n[7] Jia, Junteng, and Austin R. Benson. Neural jump stochastic differential equations. Advances in Neural Information Processing Systems 32 (2019).\n\n[8] Merton, Robert C. Option pricing when underlying stock returns are discontinuous. Journal of financial economics 3.1-2 (1976): 125-144.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their insightful comments and questions. Your feedback provided valuable suggestions and directions for improvement. Below is a brief summary highlighting the main enhancements and changes we will be making.

**Limitation and Generalization**

We will add the following concluding remarks section to clarify the limitations of our theoretical results for the proposed generator gradient (GG) estimator. Additionally, we will address the possibility of extending our theoretical results to prove the consistency of the GG estimator when the smoothness assumptions (Assumptions 1 and 3) are violated.

>The theoretical results in this paper have the limitation of requiring second-order continuous differentiability and uniform boundedness of the space derivatives of the parameters of the underlying jump diffusion. These strong conditions, which are standard in the literature of stochastic flows (cf. [Protter 1992] and [Kunita 2019] cited in the paper) to guarantee the global existence and uniqueness of the derivative processes in (3.5), are necessary to achieve the generality of the results presented in this paper.
>
> However, our generator gradient estimator often works even when coefficients are not continuously differentiable. This is true if the generator and rewards gradients are defined almost everywhere, and the derivative processes in (3.5), with almost everywhere derivatives of the SDE parameters, exist for every $t \in [0, T]$ and satisfy some integrability conditions. Examples include neural networks parameterized stochastic control with ReLU activation functions, heavy-traffic limits of controlled multi-server queues, and the Cox–Ingersoll–Ross (CIR) model. For these models, the existence and integrability of the derivative processes can be checked on a case-by-case basis, allowing the consistency and unbiasedness of the generator gradient estimator to be established. We confirm this by numerically investigating the CIR process and an SDE with ReLu drift in Appendix G.}}

As promised in the concluding remarks, we conducted additional numerical experiments using the CIR process:
$$X_\theta^x(t,s) = x + \int_t^s(\theta - X_\theta^x(t,r))dr + \int_t^s\sqrt{X_\theta^x(t,r)}dB(r)$$
and the ReLU drift SDE:
$$X_\theta^x(t,s) = x + \int_t^s(\mathrm{ReLU}(\theta X_\theta^x(t,r)) + 1)dr + \int_t^s dB(r),$$
where $x, X_\theta^x(t,s), \theta \in \mathbb{R}$. We used the proposed GG estimator and the finite difference derivative estimator (a consistent gradient estimator for both cases, though it performs poorly in high-dimensional settings) to evaluate the gradient $\partial_\theta v_\theta(0,x)$ of 
$$v_\theta(0,x) = E\left[\int_0^T X_\theta^x(t)dt\right]$$
evaluated at different $\theta > 0$. 

Numerical results in Tables 3 (CIR) and 4 (ReLU) in the attached supplement PDF indicate that in both cases, the GG estimator is consistent even if the global smoothness of the SDE parameters is violated. 

However, when $\theta < 1/2$, the CIR process is known to touch 0, making the derivative process in (3.5) only defined up to the first hitting time of 0. Therefore, the previously discussed generalized conditions don't hold. Although the numerical results in Table 3 still show consistency, we observe a surge in variance, suggesting that the GG estimator doesn't work reliably for the CIR case when $\theta < 1/2$.

**Error Analysis of Numerical SDE Schemes**

We believe it would be of significant theoretical value to demonstrate that the error of our estimator scales well with dimensions. However, it is challenging to investigate the convergence property of the algorithm in terms of the dimensions of $\theta$ and $x$. Below, we include a brief error analysis and explain why we decided against including it in the paper.

In our numerical experimentation, we use Euler schemes to approximate the SDE, which introduces a weak error (bias) of $O(\delta)$, where $\delta$ is the time-step discretization (cf. [1]).

Typical analysis of the Euler scheme (also see [1]) results in a strong error of $O(\sqrt{\delta})$. Consequently, the second moment of our estimator using the Euler scheme can be bounded by
$$E|D_\delta(x)|^2 \leq E|D_\delta(x) - D(x)|^2 + E|D(x)|^2 \leq C_1\delta + C_2(|x|+1)^{2m+4},$$
where $D_\delta(x)$ is the GG estimator from SDEs simulated using the Euler scheme, and the bound for the second term follows from Theorem 3. Here, $C_1$ would depend on $T$ and the boundedness of the space derivatives (up to second order) of $\nabla_\theta\mu_\theta$, $\nabla_\theta \sigma_\theta$, $\nabla_\theta\chi_\theta$, $\nabla_\theta\rho_\theta$, and $\nabla_\theta g_\theta$. $C_2$ depends on $T$ and the boundedness of space derivatives (up to second order) of $\mu_\theta$, $\sigma_\theta$, $\rho_\theta$, $\chi_\theta$, and $g_\theta$. 

These bounds are typically very loose in real-world examples and do not provide new insights into the validity of the GG estimator. Additionally, rigorously presenting this error analysis would require introducing a significant amount of new notation. Therefore, due to clarity and space limitations, we have decided not to include such an analysis.

[1] Kloeden, P.E., Platen, E. (1992). Numerical Solution of Stochastic Differential Equations. Applications of Mathematics, vol 23. Springer, Berlin, Heidelberg.

Meta-review: The article studies classical stochastic control in the continuous d-dimensional SDE setting and introduces a creative switching trick that reduces the simulation dimension when computing the derivative. After reading all the reviews/rebuttals/responses and the paper, the results are sound and interesting. Nonetheless, there was a significant discussion regarding the applicability and interest to the ML community of the results as the paper is dense and difficult to read with little motivations to ML. While I understand that the authors want this result to be applicable to a broader audience, some of the responsibility lies on the authors to write a paper that also appeals to the broader audience. In not doing so, the paper runs the risk that no one reads it. Please consider this and the discussion with the reviewers in the revision.
Decision: Accept (poster)