Deep learning for continuous-time stochastic control with jumps
Authors: Patrick Cheridito
Jean-Loup Dupret
Donatien Hainaut
Venue: NeurIPS 2025 poster ()
Keywords: Continuous-time stochastic control ; Jump diffusion ; Actor-critic ; Physics-informed neural networks ; Hamilton-Jacobi-Bellman equation ; Deep learning ; Bellman updating
Primary area: optimization
TL;DR: This paper introduces an iterative neural network-based approach to solve finite-horizon continuous-time stochastic control problems with jumps, when the underlying dynamics are fully known and given.

Abstract: In this paper, we introduce a model-based deep-learning approach to solve finite-horizon continuous-time stochastic control problems with jumps. We iteratively train two neural networks: one to represent the optimal policy and the other to approximate the value function. Leveraging a continuous-time version of the dynamic programming principle, we derive two different training objectives based on the Hamilton--Jacobi--Bellman equation, ensuring that the networks capture the underlying stochastic dynamics. Empirical evaluations on different problems illustrate the accuracy and scalability of our approach, demonstrating its effectiveness in solving complex high-dimensional stochastic control tasks.

Deep Learning for Continuous-Time Stochastic Control with Jumps
Patrick Cheridito Department of Mathematics ETH Zurich, Switzerland
Jean-Loup Dupret"
Donatien Hainaut LIDAM-ISBA UCLouvain, Belgium
Department of Mathematics ETH Zurich, Switzerland
{donatien. h
be
Abstract
In this paper, we introduce a model-based deep-learning approach to solve finite-horizon continuous-time stochastic control problems with jumps. We it- eratively train two neural networks: one to represent the optimal policy and the other to approximate the value function. Leveraging continuous-time version of the dynamic programming principle, we derive two different train- ing objectives based on the Hamilton-Aacoii-elllann equation, ensuring that the networks capture the underlying stochastic dynamics. Empirical evalua- tions different problems illustrate the accuracy and scalability of ap- solving complex nigh-dimensional
proach, demonstrating its effectiveness stochastic control tasks. Code is available at https://pttttt hiihhb........ commjdupret97/ Deep-Learning for- CT Ser--Stochatti-CCotttteooe
th- Jumps.
1 Introduction
A large class of dynamic decision-making problems under uncertainty can be modeled as continuous- time stochastic control problems. In this paper, we introduce two neural network-based numerical algorithms for such problems in high dimensions with finite time horizon and jumps. More precisely, we consider control problems of the form
sup T.. L S(t, (1, oz)dt + F(X9)
for finite horizon T>> O, where the supremum over predictable control processes a = (at)osesT taking values a subset A C RR, The controlled process X evolves subset D C Rdd according to
dxx = Btt, at) ot,, X; d.W.W, (t (dz, dt), XØ
D, (2)
for an initial condition E D. an n- n-dimensional Brownian motion W and a controlled random
measure No on E x R_, with E R' \ {0), and suitable functions 8: [0, T] x x A Rd pdx= and 7: [o, x Ex A - Rd The functions f: [0, Dx
a: [0, 2 D x and F: D + R model the running and final rewards, respectively. We assume the controlled random measure is given by NO (B x [0, Σ 1 ((, for measurable subsets B S E, where M is Cox process with stochastic intensity of the form Xf=, av)) and Z1, Zz.... random vectors such that, conditionally on a, the random elements W, Ma and Z1, Zz,... are independent. Our goal is find an optimal control a* and the corresponding value of problem
i.i.d. E-valued
"Corresponing author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(I). In view of the Markovian nature of the dynamics (2), we work with feedback controls of the form a = a(t, X for measurable functions a: [0, T) D A and consider the value function V: : [ x D + IR given by
J. f(s, Xss F(x7) X =
((t, sup E a
Under suitable assumptions V is the unique solution of the following Hamitton-Jacobi-eellman (HJB) equation
(tV sup (1, V.a) = 00
=
aAA
for the Hamiltonian H (0, x Dx
R given" by
a(t, V. = f(t, 3d (t, x a) V ((, + ! Ir (t,,, (t. 2
X(1, x. a) B[V (1, x 7(,,,, Z2, a)) =)),
where V.V, V2V denote the gradient and Hessian of with respect to æ. Our approach consists in iteratively training two neural networks to approximate the value function V and an optimal control a" attaining V. has the following features:
It yields accurate results for high-dimensional continuous-time stochastic control problems in cases where the underlying system dynamics are known.
It can effectively handle combination of diffusive noise and random jumps with controlled intensities.
It can handle general situations where the optimal control is not available in closed form but has to be learned together with the value function.
It approximates both the value function and optimal control at any point (f,x)= E [0,T) x D.
While methods like finite differences, finite elements and spectral methods work well for solving partial cnitero->xiiferentii equations P(I)DEs in low dimensions, they suffer from the curse of dimensionality and, as a consequence, become infeasible in high dimensions. Recently, different deep learning based approaches for solving high-dimensionaal PDEs have been proposed (Raissi et al., 2017, 2019; Han et al., 2017, 2018; Sirignano & Spiliopoulos, 2018; Berg & Nyström, 2018; Beck et al., 2021; Lu et al., 2021; Bruna et al., 2024). They directly be used to solve continuous-time stochastic control problems that admit an explicit solution for the optimal control in terms of the value function since in this case, the expression for the optimal control be plugged into the HJB equation, which then reduces to a parabolic PDE. On the other hand, the optimal control not available in closed form, cannot be plugged into the HJB equation, but instead, has to be approximated numerically while at the same time solving a parabolic PDE. Such implicit optimal control problems longer be solved directly with one of the deep learning methods mentioned above but require specifically designed iterative approximation procedure. For low- www-dmensional problems with non- explicit optimal controls, standard approach from the reinforcement learning (RL) literature is to use generalized policy iteration (GPI), a class of iterative algorithms that simultancously approximate the value function and optimal control (Jacka & Mijatović, 2017: Sutton & Barto, 2018). Particularly popular are actor-critic methods going back to Werbos (1992), which have separate memory structure represent the optimal control independently of the value function. However, classical GPI schemes become impractical in high dimensions as the PDE and optimization problem both have to be discretized and solved for every point finite grid. This raises the need for meshfree methods to solve implicit continuous-tine stochastic control problems in high dimensions with continuous action space. Several local approaches based on a dime-discretization of (2) have been explored by e.g. Han & E (2016), Nüsken & Richter (2021). Huré et al. (2021), Bachouch al. (2022), Ji et al. (2022), Li et al. (2024), Domingo-Eminn Enroo--Ercchhhhi et al. (2024a), Domingo-Ennen Ennich et al. (2024b). Alternatively, (deep) RL methods can be used, for instance. Q-learning type algorithms such DQN (Mnih et al.. 2013) or C51 (Bellemare et al., 2017), see also Wang et al. (2020), Jia & Zhou (2023), Gao et al. (2024), Szpruch et al. (2024); actor critie approaches such DDPG (Lillicrap et al., 2019), SAC (Haarnja
Psee e.g. Soner (1988)
'By 12 we denote the set of all functions in 2 x Ie such that the expectation in (4) is finite for all fr and

---

et al., 2018), A2C/A3C (Mnih et al., 2016), PPO (Schulman et al., 2017) or TRPO (Schulman et al., 2015). However, these RL algorithms are model-free, that is, they do not explicitly take into account the underlying dynamics (2) of the control problem (1) but instead, solely rely sampling from the environment. As result, they are less accurate cases where the system dynamies are known (see Figure 3 below). On the other hand, the difficulty of solving high-dimensional PDEs is further exacerbated if the system dynamics includes stochastic jumps since in this case, the HJB equation (3) requires the computation of the jump expectations E (t, + 7t, x, Z1, a)) for every space-time point (t, .) sampled from the domain.
In this paper, we introduce a deep model-based approach for stochastic control problems with jumps that takes the system dynamics (2) into account by leveraging the HJB equation (3). This removes the need to simulate the underlying jump- cump-diffusion (2) and, as a result, avoids discretization errors. Our approach combines GPI with a PIDE solving method. approximates the value function and optimal control in an actor-critic fashion with two neural networks trained iteratively on sampled data from the space-time domain. As such, it has the advantage that it provides global approximations of the value function and optimal control available for all space-time points, which can be evaluated rapidly in online applications. We develop two related algorithms. The first one, GPI-PINN, approximates the value function by training a neural network to minimize the residuals of the HJB equation (3), following a physics-inspired neural network (PINN) approach (Raissi et al., 2017, 2019) while leveraging Proposition 3.1 below avoid the direct computation of the gradient V and Hessian V in the Hamiltonian (4). GPI-PINN can be viewed as an extension of the method proposed by Duarte et al. (2024) adapted to a finite-horizon setup with time-dependence and terminal condition in the HJB equation, leading to time-dependent optimal control strategies and value function, see also Dupret& & Hainaut (2024). It works well in high dimensions for control problems without jumps in the underlying dynamics (2) (7 = 0). but becomes inefficient in the presence of jumps as it requires the computation of the jump-expectation E (t, + 7(1 x, Z1, a)) in every iteration of the algorithm. To address this, our second algorithm, GPI-CBU, relies on a continuous-time Bellman updating rule to approximate the value function, thereby cireumventing the computation of gradients, Hessians and expectatens altogether. This makes it highly efficient for -ighdiimensiona stochastic control problems with jumps, even when the control is not available in closed form.
a)) at numerous sample points (1,
We illustrate the accuracy and scalability of our approach in different numerical examples and provide comparisons with popular RL and deep-learning control methods. Proofs of theoretical results and additional numerical experiments given in the Appendix.
2 General approach
Let a: [0, x D + A be a feedback control such that equation (2) has a unique solution X and consider the corresponding value function
(t x) E J.""
Isds F(X9)
X x
(t, E (0, T x D.
Under appropriate assumptions, one obtains the following two results from standard arguments*.
Theorem 2.1 (Feynman-Kan Formula). V satisfies the PIDE
0, (t,z), + H(t, V .((t,))) = 0, (T(T,z) = F()).
Theorem 2.2 (Verification Theorem). Let Vn C( T] x D) be solution of the HJB equation (3) such that there exists a measurable mapping à (0, x D - A satisfying
a(t, x) E arg max H (t, for all (t, (0, aEA
and the controlled jump-diffusion equation (2) admits a unique solution for each initial condition D. Then u V and â is an optimal control.
Based on Theorems 2.1 and 2.2, we iteratively approximate the value function V and optimal control a" with neural networks V9: [0, 2] D + R and as: : [0, T] x D + A. For given agg we train Vo
'see e.g. the arguments in the proofs of Theorems 1.3.1 and 2.2.4 in Bouchard (2021).
Using a O -cctiivvaionn function in the network Vo ensures that it belongs to (20 Hd).
3

---

so as to solve the controlled HJB equation
VeVe + Htt, a, Ve, as, = 0, Vo(T, = F
(5)
while for given Vo, Ore is trained with the goal to maximize the Hamiltonian H(,, Vo, (1,(1, x)).
In the following. introduce two different training objectives for updating the value network Ve, leading respectively to the algorithms GPI-PINN and GPI-CBU. GPI-PINN uses PINN-type loss together with trick adapted from Duarte al. (2024) to bypass the explicit computation of gradients and Hessians, whereas GPI-CBU relies on a continuous- Bellman updating rule with expectation preee version of the Hamiltonian, thereby also avoiding the computation of the jump-expectations in (4).
3 GPI-PINN
GPI-PINN, described in Algorithm below, relies on a PINN approach to minimize the residuals of the controlled HJB equation (5) in the value function approximation step. To avoid explicit computations of the gradient (t, and Hessian Vo (t, x), which appear in the Hamiltonian (4), the following trick, adapted from Duarte al. (2024).
Proposition 3.1. Consider a function v E V together with pair (1,x) (0, x D. Define the function vp R R by
(h))
Σ
11
1+
h2
h x.
01,
h²
3(,, x, 2.)))
2n
V2
2n
where oi x, a) is the 2th column of the d x matrix o (t, Then.
* (0) = 0,v(t, x) + (t, (t, x, Vou(t, +
2 Tr at, (()t,x)
Proposition 3.1 makes it possible to replace the computation of gradients and Hessians of by evaluating the univariate function (0), the cost of which, using automatic differentiation, a small multiple of n cos)(v).
To formulate GPI-PINN, we need the extended Hamiltonian
H(t,I,v,, == 0,v(t,z) (,,
(6)
which by Proposition 3.1, can be written as
H(t, x, a) = (0) + =t,,,, ( + 21, utt,z)]
In Algorithm 1, we simplify the notation by using H(t, x, 0, = ao, x)).
The loss function Li in (7) consists of two terms. The first represents the expected PIDE residual in the interior of the space-time domain with respect to a suitable measure on [0, T) x D. The second term penalizes violations of the terminal condition according to a measure v on D. Hence, Li measures how well the function Ve satisfies the controlled HJB equation (5) corresponding to therefore to find parameter vector 0 such that the value network VA minimizes the error L1 (0, (A))). We do this with a mini-batee stochastic gradient method which updates the measures j and v according to the residual-basee adaptive distribution (RAD) method of Wu et al. (2023), as it is known to significantly improve the accuracy of PINNs. In Step 2, we minimize Ls (0(k+1), () with respect to σ. This corresponds to choosing the control as so to maximize the extended Hamiltonian H (or equivalently H); see Duan et al. (2023), Dupret & Hainaut (2024) and Cohen et al. (2025) for theoretical convergence results supporting this approach.
control ag. In every epoch k, the goal in Step
Since the expectations in LL, Llg and the extended Hamiltonian (6) are typically not available in closed form, we replace them by sample-based estimates. First, we estimate H(tt, xr, 0, Φ) with il(t, x. f. σ) by approximating the jump-expectation
EVo(t, + 7(t, Z1, az(tz))))
Volt, 1 t
(9)

---

Algorithm GPI-PINN
Initialize admissible weights g(0) for V and (00) for ag. Choose proportionality factors f., and set epoch & = 0. repeat
Step 1: Update the value network Va(+t for given control agw) by minimizing the loss
g(k+1) = arg min L1 (i ()(1),
where
()) = p H. (1, 1,, + B (Vo(T, F(x:))²
Step 2: Update the control network ag(+t1) for given value network Vou+n) by minimizing
the loss
((++1)) arg min (g(k+)1) .6), ,)),
where
L2 0) M1 H I, 0,
(8)
1: t ++ until some convergence criterion is satisfied. return Vycs and (g(A)) and set +- k.
for points (z)) in E' sampled from the distribution Z of Z1. every time step by
Then, we approximate L1 and L2 in
Mi
M2
(k)))
Σ
AA(mm Dm, 0 o(()))
+
&
Σ
(Vo(T, ym) - ()m)))
M
M2 m=1
m=1
and
Mi
((((+11) x)
1
Σ
Hl
tm Tmgy g(R++1) .))
M1
m=1
where (m,
M1 m= E [0, 7) D and (ym M Fmm every epoch ......
M2
E D are sampled from μ and v, respectively. In
......,. ki 1, we initialize 0/²)
= o(k) and make N, gradient steps
Mi
o(2) = 0!() -m mi (0 ()) 0000)
20 Mi
Fl Fm, a (k) Vo Po Im, o(k) (b())
m=1
2m E2
Σ
(T) vm) F(Um)) VoV. (TV Um). (10)
M2
i i= 0, N1 1, to obtain g(++1)
1//
Then, we initialize 600 = ((0)) = and perform No gradient
steps
Mr
(k)
0,² V Eg (K)) = ((()
1/2
V R (tm, I' g(++1) (
(II)
0+1
M1
i 0, O...., N₂
1, to get (f(--1) = (b)
GPI-PINN yields global approximations Vo(k.) and agik.) of the value function and optimal control on the whole space-time domain [0, 7] D. By using Proposition 3.1, it avoids the computation of the gradients and Hessians appearing in the Hamiltonian (4). However, it still has two drawbacks that make it inefficient for high-ddmensional control problems with jumps. First, has to approximate the jump-expectations E Ve Imm + 7 Im Z1, 0.g (fm, xm)]]] for all sample points (Im Im), m 1,..., M1. in each of the gradient steps (10)(11) and all epochs Secondly, since the Hamiltonian is already second-order integro- differeiifer operator, the gradient steps V H in (10) require the computation of third order derivatives, which is numerically costly.
1,..., k, see (9).
5

---

4 GPI-CBU
GPI-CBU addresses the shortcomings of GPI-PINN by using value function updating rule based on the expectation-free operator G x D
given by
Gcll, 1, U a) = v(1, x) ([0,o((,2) + ((, + (1, V20,,,)))u0(,.2 (4)
2 TT Tto 2. (t.z, A(t, x, a) (u , 1t, v(t, x))]
for scaling factor < ER.
Proposition 4.1. Let Xo be a solution of the jump-diffuee mimpdiffusion equation (2) corresponding to feedback control For given t E [). let Y, be a D-valued random variable independent of Z, such that E G Y1, Z1, Va, a(t, Y.)) co. Then (1, g(Y)) for the Borel measurable function
[0, x D A with associated value function v (),2 x D).
g D + I. minimizing the mean squared error
[((o Ge(t, Y1, 21 Vo, at, (.))
Proposition 4.1 suggests to update the value function parameters according
g(++1) arg min I. /," (Volt, Yi) Gc(t, Y,, Zi, Vaus a(t, Yi)) dt.
(12)
By adding a penalty term enforcing the terminal condition, we obtain the recursive scheme
g(++1) arg min Se(h)) (0)
(13)
for the loss
( =0)= (0) (Vo(l.2) - Gc(,, 2,
2, ((0) ,(a1)) + +z (Vo(T - -(x)))
where we use the notation Gz 2, Gz(t, x, 2, Vo, ap(t, To implement (13), approximate (2(h((0
(0) with
M1
M2 (Vo(T, F(ym)) (14)
(0)
E1
Σ
(bm; Cc (1m, Im, Zm, ((k) ,0()))
2
&
Σ
M1 mmm1
M2
for (fms Im Zm m- E (0,T) D x sampled from μ1 8 Z and Ma
Mr
E D sampled from v.
In every epoch Is 0 ..... k. initialize g6*²) (k)) and perform N1 gradient steps
Mr Σ
((0) ((0)
(o!t)) gC
2
(T Mm.) ((ym)) Vol VoVins(1
(15)
Mi
E
Voli(tm; am)
Ge (tm, Tmn &m, 06*) )()) Vo Vo
M1
m=1
0,.... N1 1, to obtain ((++1) (l) To update the control network ag, introduce the operator
(1t,z,2,0,00) = Vo(o f(t, (t, x)) + T (t, V.
+ +2 Tr oo (t, as (z) a(t, (Vo + ((t, at() Vo(t,
which is an expectation-free version of the extended Hamiltonian that, instead of the jump-expectation (o,, + >t,,x, Z1, ()), only contains single jump Vo(t, (t, 2, 2, E. GPI-CBU updates the parameters of the control network ag according to
((++1) arg min L (k+ (Ø)
'By (22), the update rule (12) corresponds to Vo(k = 7 V (k)) := Vyua + CHL(, Vaccs, a(-)) for the continuous-ting Bellman updating (CBU) operator 7 associated with the feedback control a and fixed point yaya Va

---

for the loss
L (0) (0)
(t.r,,)) =002 g(t, 2, 2, 1(k+t)) .6) (t H(t, I, g(++t) .0).
We hence minimize the sample estimate
M2
2
(0) (2)
Σ
(( Um 2m, glk+1), .6),
(16)
M1 m=1
by setting 60 2b(h) and making N2 gradient steps
M1
(k) ; 1/2 V
1 (0 ( = Φ;
(k)
+
72
(1m Imm Jmy 0(++1) ,0!
(17)
M
12
i = 0, N2 1, to obtain ((++1) (0)
(k)
N2'''
Algorithm 2 GPI-CBU
Initialize admissible neural weights ((0) for Vo and (i(0) for Otg Choose learning rates vn, 7/2 proportionality factors Eι, E2 and numbers of gradient steps Ni, N2. Set epoch k = 0.
repeat
Step 0: Generate M₁ sample points (tm: Tma 2m) E [0, x D x E from @ Z and M2 sample points ym D from v. Step 1: Update Vot+ta by minimizing the loss (14)
((++1) arg min k0th) (0),
with N1 gradient steps (15).
Step 2: Update Ogga+11 by minimizing the loss (16)
(k+11) arg min 21 (0),
with N2 gradient steps (17).
k 4- A ++ 1
until some convergence criterion is satisfied. return Vous and O'g(c) and set k; <- k.
Like GPI-PINN, GPI-CBU leverages Proposition 3.1 to bypass the computation of the gradients V.Vo and Hessians Vo. In addition, it avoids the costly computation of the jump-expectations (9) of Vo at each sample point (Im, Tm). Also, it does not have to compute third-order derivatives like (10) when updating the value network since only V Vo is needed in (15). This is consequence of the fact that the value update rule (15) of GPI-CBU is recursive as Ge depends on the value weights g(k) computed in the previous epoch in contrast to the residual-based GPI-PINN. On the other hand, since GPI-PINN averages over different jumps in each update, it exhibits more stable convergence than GPI-CBU; see Figure below. The proportionality factors E1, &2 and the scaling factor Ç are hyperparameters. & and ε2 can be fine-tuned following Wang al. (2022). While Proposition 4.1 holds for any scaling factor Ç E R, its choice affects the numerical performance of GPI-CBU. In the numerical experiments of this paper, convergence speed and accuracy of the improvements in GPI-CBU. On the other hand, negative scaling factors failed to converge to the true solution in all our experiments with exploding losses
set 1 as it provides a good trade-off between
Gith) and 27th) after only a few epochs. Alternatively, one could consider an adaptive scaling factor Co >>> depending the epoch k. More details on hyperparameter fine- mne-tuning given in Appendix B. Finally, we note that the policy updating rule of GPI-CBU (17) is equivalent to that of GPI-PINN (1I), except that it circumvents the computation of the jump-expectations in H. since it uses the expectation-free operator G.
5 Numerical experiments
In our numerical experiments, we use the Deep Galerkin Method (DGM) architecture of Sirignano & Spiliopoulos (2018) for both the value and optimal control networks as it has been shown to
7

---

empirically improve PINN performance. Details of the network design and hyperparameters are given Appendix
5.1 Linear-quadratic regulator with jumps
We first consider the linear-quadratic regulator (LQR) problem with jumps
caalla Lo allaal12d ds c2llx
inf a
(18)
where the infimum is over dddimensiooall predictable processes (at and X a d-dimensional process with controlled dynamics
Mr Zg. Xo = x E Rd
dx = =c,dt + BdW, +
(19)
3=1
for d x d matrix B, a d-dimensional Brownian motion W, Cox process M with intensity XØ = Ai + d2]los1] 1/ for constants A1, A2 2 and independent i.i.d. zero-mean square integrable d-dimensional random vectors Zj with [B]] 2, = U. 1.2....
Problem (18) is the general form (1) if instead of the infimum, one considers the supremum of the negative of the expectation. It can be reduced to oiedimensionall ODE which has closed form solution in the special case A2 = 0 and admits a very precise numerical solution via a standard ODE solver, such as Runge- Kutta, if A2 > 0 (see Appendix C). This provides highly accurate reference solutions V and o for the value function and optimal control.
Figure 1 compares the performances of GPI-PINN and GPI-CBU on 10-dimensional (d = 10) LQR problem of the form (18) with T 1, B = Ia, e₁ = 1, C2 = 1/4 without jumps (A = A2 = 0) and with jumps (A, 0, A2 2 and d-dimensional standard normal jumps Zj). It shows mean absolute errors of Voc. with respect V given by MAE, = Ar ΣM [Vano(ts, - #t), on a test set of size M uniformly sampled from [o, 1] x [-2.5, 2. together with runtimes as functions of the number of epochs k. It can be seen that GPI-PINN and GPI-CBU exhibit similar convergence in the number of epochs. The residual-based approach of GPI-PINN makes it more stable than GPI-CBU; see also Baird (1995). On the other hand, GPI-CBU has lower computational cost. This already the case without jumps (left plot of Figure 1) since it avoids the numerical evaluation of third-order derivatives but becomes much more significant with jumps (right plot of Figure 1) it also circumvents the numerical integration of the jumps.
035
TPLRME withor
PID
303
4422
45000
615
2000
0.3
010
900
088
Figure 1: Comparison of MAEy (blue) and runtime in seconds (green) of GPI-PINN (solid lines) and GPI-CBU
(dashed lines) for a 10-dimensional LQR problem (18) without jumps (left) and with jumps (right).
Figure shows results of GPI-CBU for the same LQR problem with jumps in d = 50 dimensions. While the high dimensionality renders GPI-PINN infeasible, GPI-CBU achieves high accuracy in the approximation of the value function as well as the optimal policy. Additional results for up to 1 150-dimensional LQR problems reported in Appendix C.
8

---

136
13.4
Value Function
Optimal Control
0.00
12.6
-0.05
12.4
Reference GPLCBU
122
-0.10
GPICBU
Figure 2: Value function V(0,=)) .x) (left) and first component of the optimal control ai (0. (right) for (21, .,......................... 0) with E [-2.5, 2.5) for a 50-dimensional LQR problem with jumps. Orange dotted lines: numerical results GPI-CBU with #1 standard deviation given by orange shaded area. Blue lines; reference solutions V and a".
Figure shows the accuracy GPI-CBU the 10- dimensional LQQ problem with jumps from the right panel Figure compared with two popular model- free RL algorithms PPO and SAC as well as the model- based discrete-tingg time approach Han & E (2016) applied to time-i dinediscretization of the state dynamics (19). It seen that this setup, PPO and SAC cannot compete with the model-based approaches since they do explicitly use the dynamics (19) but instead only rely on sampling from the environment. The method of Han & (2016) outperforms PPO and SAC but does not achieve the accuracy GPI-CBU due discretization rors and since does not generalize well to unseen points in the test set. Trying cover the space-time domain well, we ran the method of Han & E (2016) from sev- randomly sampled starting points zu E But being a local method, it tends to learn the optimal control only along the optimal state trajectories X1 JoSeST. which results in poor performance in parts of the space- domain that are not explored well. Additional results are discussed in Appendix C.
Log MAE
1000
2000
3000
4000
5000
Epoch (Beration)
Figure 3: log MAEv of different deep-learning methods for 10-dimensional LQR problem with jumps.
5.2 Optimal consumption-investmees with jumps
As a second example, we consider an economic agent who consumes at relative rate a and invests in financial assets according strategy? (mI....,,
m²" maximize
L
E
ps un ds - UYY))
(20)
for two utility functions u, U: 14, + R. where Y, is the wealth process evolving like
Mi
aY YE
a Fely - rt) - - dt
of aw; + d <E(es
rr
m
(21)
for a stochastic interest rate Tt, expected return rates μ², stochastic volatilities of, an n-dimensional Brownian motion W with correlation matrix Σ, Cox processes M with stochastic jump intensities X and random jump variables Zj. We consider strategies the form and c(t, ae, de, Y and
(1, Ct, Xo, Tt, Y2). The problem has d = 2n + 2 state variables, consisting of of X rt. ya and
"T describes fraction of agent's total wealth held in the th asset at time In numerical experiments, we choose CRRA utility functions.

---

n+1 decision variables Ct, Tz. i 1..... n. In general, the corresponding HJB equation, given in (35) in Appendix D.2. does not have an analytical solution. But it can be seen Figure 4 that GPI-CBU converges numerically for = 25 financial assets. More details about this consumpiion-investmentt problem with stochastic coefficients are provided in Appendix D.2.
In Appendix D.I, we consider simplified version of the problem where the coefficients dy X Pr are constant. This case be reduced one-dimensional ODE that can be solved with a standard Runge-Kutta scheme to obtain reference solutions. GPI-CBU yields numerical solutions that are virtually indistinguishable from the Runge Kutta results.
Boundary 44000
Intertor Loss
25
42000
40000
Loss
15
36000
34000
32000
0-
30000
200
400
600
800
1000
1200
1400
200
400
600
800
1000
1200
1400
Epoch.k
(0(+=)) (left) and (0 (2)) (right) of GPI-CBU functions of the epoch le for consumptionninvesttee investment problem the form (20) (21) with n 25 financial assets. The blue curve in the left plot represents the interior loss of (Ppk) and the orange curve its boundary part; see (14).
Figure 4: Training losses 2
6 Conclusions, limitations and future work
In this paper, we have introduced two iterative deep learning algorithms for solving finite-horizon stochastic control problems with jumps. Both use an actor-critic approach and train two neural networks to approximate the value function and optimal control, providing global solutions over the entire space-time domain without requiring simulation or discretization the underlying state dynamics. Our first algorithm, GPI-PINN, works well for high-dimensional problems without jumps but becomes computationally infeasible in the presence of jumps. The second algorithm. GPI- CBU, leverages an efficient expectation-fiee updating rule based on Proposition 4.1 which makes it particularly well-suited for high-dimensional problems with jumps. Both algorithms are model- As such, they outperform model-free RL methods in cases where the underlying state dynamics are known. The accuracy and scalability of the two algorithms has been demonstrated in different numerical examples.
limitation of our approach lies in the need to know the underlying dynamics of the state process, which are not always available in real-world applications. While it is reasonable to assume that physical systems obey known laws of motion, dynamics in economics and finance typically need to be inferred from data. However, in such cases, they can be learned in preliminary step using e.g. recent model-learning algorithms such as Brunton et al. (2016) or Champion et al. (2019). Once an approximate model has been learned from data, one of the proposed algorithms, GPI-PINN or GPI-CBU, can be applied to efficiently solve the resulting stochastic control problem.
Acknowledgments
Financial support from Swiss National Science Foundation Grant 10003723 gratefully acknowl- edged. We thank the reviewers for their helpful comments and constructive suggestions.
References
Bachouch, A., Huré, C., Langrené, N., and Pham, H. Deep neural networks algorithms for stochastic control problems on finite horizon: numerical applications. Methodology and Computing in Applied Probability, 24(1):143-178, 2022.
10

---

Baird, L. Residual algorithms: reinforcement learning with function approximation. International Conference on Machine Learning, 12:30-37, 1995.
Beck, C., Becker, S., Cheridito, P., Jentzen, A., and Neufeld, A. Deep splitting method for parabolic PDEs. SIAM Journal on Scientific Computing, 4335)44115,,33154, 2021.
Bellemare, M. G., Dabney, W., and Munos, R. A distributional perspective on reinforcement learning. International Conference on Machine Learning, 34:449-455 458, 2017.
Berg, J. and Nyström, K. A unified deep artificial neural network approach to partial differential equations in complex geometries. Neurocomputing, 317:28-41 2018.
Bouchard, B. Introduction Stochastic Control of Mixed Diffission Processes, Viscosity Solutions and Applications in Finance and Insurance. Ceremade Lecture Notes, 2021.
Bruna, J., Peherstorfer, B., and Vanden-Eijnden, Neural Galerkin schemes with active learning for high-dimensions evolution equations. Journal of Computational Physics, 496, 2024.
Brunton, S. L., Proctor, L., and Kutz, N. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the National Academy of Sciences, 113(15)33932-3337, 2016.
Champion, K., Lusch, B., Kutz, J. N., and Brunton, S. L.. Data-driven discovery of coordinates and governing equations. Proceedings of the National Academy of Sciences, 116(45):2245-22451 2019.
Cohen, S. N., Hebner, J., Jiang, D., and Sirignano, J. Neural actor-critic methods for Hamilton- Jacobi-Bellman Bellman PDEs: asymptotic analysis and numerical studies. arXiv Preprint 2025.
Domingo Enrich, C., Drozdzal, M., Karrer, B., and Chen, R. T. Adjoint matching: fine-tuning flow and diffusion generative models with memoryless stochastic optimal control. arXiv Preprint 2409.08861, 2024a.
Domingo-Earich, C., Han, J., Amos, B., Bruna, J., and Chen, R. T. Stochastic optimal control matching. Advances in Neural Information Processing Systems, 37:1
2024b.
Dormand, J. R. and Prince, J. A family of embedded Runge- Kutta formulae. Journal of Computa- tional and Applied Mathematics, 6(1):19-26, 1980.
Duan, J., Li, J., Ge, Q. Li, S. E., Bujarbaruah, M., Ma. F., and Zhang, D. Relaxed actor-critic with convergence guarantees for continuous-time optimal control of nonlinear systems. IEEE Transactions Intelligent Vehicles, 35)33329-3331, 2023.
Duarte, V., Duarte, D., and Silva, D. H. Machine learning for continuous time finance. The Review of Financial Studies, 37(11):3217-32) 2024.
Dupret, J.-L. and Hainaut, D. Deep learning for high-dimensional continuous-tin time stochastic optimal control without explicit solution. Technical report, Université Catholique de Louvain, Institute of Statistics, Biostatistics and Acturial Sciences, 2024.
Durrett, R. Probability: Theory and Examples. Cambridge University Press, fifth edition, 2019.
Gao, X., Li, L., and Zhou, X. Y. Reinforcement learning for jump simp-diffusions with financial applica- tions. arXiv Preprint 16449, 2024.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: off off-policy maximum entropy deep reinforcement learning with stochastic actor. International Conference on Machine Learning, 35: 1861-1870, 2018.
Han, and E, W. Deep learning approximation for stochastic control problems. arXiv Preprint 1611.07422, 2016.
11

---

Han, J., Jentzen, A., and E, W. Deep learning-based based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations. Communi- cations in Mathematics and Statistics, 344):349-380, 2017.
Han, J., Jentzen, A., and E, W. Solving high- -igh-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 115(34)88505-88)) 8510, 2018.
Huré, C., Pham, H., Bachouch, A., and Langrené, N. Deep neural networks algorithms for stochastic control problems on finite horizon: convergence analysis. SIAM Journal on Numerical Analysis, 59(1):525-55) 2021.
Jacka, S. D. and Mijatovic, A. On the policy improvement algorithm in continuous time. Stochastics, 89(1):348-335 2017.
Ji, S., Peng, S., Peng, Y., and Zhang, X. Solving stochastic optimal control problem via stochastic maximum principle with deep learning method. Journal of Scientific Computing, 93(1):30, 2022.
Jia, Y. and Zhou, X. Y. q-learning in continuous time. Journal of Machine Learning Research, 24 (161):1-61, 2023.
arXiv preprint Kingma, D. and Ba, J. Adam: method for stochastic optimization. arXiv: 2014.
X., Verma, D., and Ruthotto, L. A neural network approach for stochastic optimal control. SIAM Journal on Scientific Computing, 46(5):C535 C556, 2024.
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. arXiv Preprint 1509.02971, 2019.
Lu, L., Meng, X., Mao, Z., and Karniadakis, G. E. Deepxde: deep learning library for solving differential equations. SIAM Review, 63(1):208-228, 2021.
Mnih, V., Kavukcuoglu, K., Silver, D., Alex, G., Antonoglou, I., Wierstra, D., and Riedmiller, M. Playing Atari with deep reinforcement learning. arXiv Preprint 1312.5602, 2013.
Mnih, V., Puigdomènech Badia, A., Mirza, M., Graves, A., Lillicrap, T. P.. Harley, T., Silver, D., and Kavukcuoglu, K. Asynchronous methods for deep reinforcement learning. arXiv Preprint 1602.01783, 2016.
Nüsken, N. and Richter, L. Solving high-dimensional Hamilton-Jaute Jacobi-Bellman PDEs using neural networks: perspectives from the theory of controlled diffusions and measures on path space. Partial Differential Equations and Applications, 2(4):48, 2021.
Øksendal, B. and Sulem, A. Applied Stochastic Control of Jump Diffusions. Springer, third edition, 2007.
Raissi, M., Perdikaris, P., and Karniadakis, G. E. Physics informed deep learning (Part 1): data-driven solutions of nonlinear partial differential equations. arXiv preprint arXiv:1711.10533 2017.
Raissi, M., Perdikaris, P., and Karniadakis, G. E. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 201,
Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. Trust region policy optimization. International Conference on Machine Learning, 32: 1889-1897, 2015.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiy Preprint 1707,06347, 2017.
Sirignano, J. and Spiliopoulos, K. DGM: A deep learning algorithm for solving partial differential equations. Journal of Computational Physics, 375:1339-1364 2018.
Soner, H. M. Optimal control of jump-markov processes and viscosity solutions. Stochastic Differential Systems, Stochastic Control Theory and Applications, pp. 501-511. Springer, 1988.
12

---

Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018
Szpruch, L.. Treetanthiploet, T., and Zhang, Y. Optimal scheduling of entropy regularizer for continu- oustime linear-quadratic reinforcement learning, SIAM Journal on Control and Optimization, 62 (1):135-166, 2024.
Wang, H., Zariphopoulou, T., and Zhou, X. Y. Reinforcement learning in continuous time and space: stochastic control approach. Journal of Machine Learning Research, 21(198):1-34, 2020.
Wang. S., Yu, X., and Perdikaris, P. When and why PINNs fail train: neural tangent kernel perspective. Journal of Computational Physics, 449:110766 2022.
Werbos, P. Approximate dynamic programming for real-tine wul-timee control and neural modeling. Handbook of Intelligent Control: Neural, Fuzzy and Adaptative Approaches, pp. 493-526, 1992.
Wu, C., Zhu, M., Tan, Q. Kartha, Y., and Lu,
comprehensive study of non-adaptive and sssiuual-bsees adaptive sampling for physics-inforie informed neural networks. Computer Methods in Applied Mechanics and Engineering, 10671, 2023.
13

---

A Proofs
A.1 Proof of Proposition 3.1
2(, the second-order derivative of Denoting 0:(+) u(t + 2n x (2,, + 1/2
2
2n
(h) Σi (/(()) is given by
(1()) Σ Divi(t) ne oiv: +2 h
h²
( (o)lt,2,,)
h @ Veaue())
V2
3 3 (), [
t - Piol)) 20lt,,2,0)
h
V2
V2
Evaluating it at 0 gives
= -, (t,,, ((x,t= _ (t, ()),,,)))(t
which proves the proposition.
A.2 Proof of Proposition 4.1
Since Y, independent of Z1, one obtains from the definition of Gę that
(t, Y, Zi, Va, (t, YJ) = = = + (t f(t,z,
+ B (tt 2 F00 (t..t
+ x1 (((tt)))
VV*(tz)) H(,,,,, vo, (*tt,,,
(22)
2),
where the last equality follows from Theorem 2.1. On the other hand, well known that
EI Y1, Z1, Vo a((,, Y.)) Y.] = g(Y))
for the Borel measurable function g: D - R minimizing the mean squared error
E (o(Y)) Gt Y,, Zi, Va, at, V",,VV"", Y())²
see e.g. Theorem 4.1.15 of Durrett (2019). This completes the proof.
B DGM Architecture
(l,x)
oy (W) (c.x) +b)
DGM Layer
DGM Layer
DGM Layer
S1
gh+1
on (W- 5d+1 +
V/ xt.
Figure 5: DGM architecture of the value neural network with L 3 (f.e. hidden layers).
14

---

Each DGM layer in Figure 5 in the value network Ve is of the following form
s σv (WI (t.))
z = ov (t,z)) + Wtd 5t + )(d), ( -=.,... L
{= 1 L Rº ov (UNF. (1,2) (1 + Wnd 5 +b0,, {= ll.,., Lu, H' = σv (U (t,x) - whe (s o RR) + +bh),, = 1....,,
G ov (U9 (t,z) + W9r 8 + -10))
st+1 = - G1 @ H + z ② o (=
vt( o... (W s + b)
where the number of hidden layers I. + 1, denotes matrix multiplication and @ element-wise multiplication. The DGM parameters of the value network are
0 = (w = b¹, (U Wa. b tm (voe, Woe (..))) (Ure, w B L 8=1 (Uhee whe b..9 L 8=1 W; W.b..
The number of units in each layer is N. ov RV + IR is twice-differentiable element-wise nonlinearity and Vou. -V + V is the output activation function, also twice-differentiable, where V C IR is chosen as satisfy possible restrictions on the value function's output, resulting from the form of the stochastic control problem (e.g. non-negative value function). The control network is designed following the same DGM architecture with a R A. Throughout the numerical examples of the paper, we use L = 3 with N = 50 neurons in each of the DGM layers, see Figure In the value network, we use tanh for the activation function øy and softplus for V For the control network, we adopt tanh for da. while as output activation aa.. we choose the identity in Example 5.1 and softplus in Example 5.2.
Unless stated otherwise, we use a number of sample points M1, M2 equal to 256, a number of gradient steps N1, .2 equal to 64 and a maximum number of epochs k. 1500. The network parameters are updated using Adam (Kingma & Ba, 2014) with constant learning rates 1/, 122 equal 0.001. Both GPI-PINN and GPI-CBU are fairly robust with respect these choices. In our setting. the most critical hyperparameters for convergence are the proportionality factors {1, Ez, which we determined using the approach of Wang et al. (2022), together with the scaling rate Ç (specific to GPI-CBU). In our experiments, we set speed and accuracy of the improvements in GPI-CBU. Alternatively, one could consider an adaptive scaling factor S& > 0 depending on the epoch k. On the other hand, negative scaling factors resulted
as it provides a good trade-off between convergence
in non-convergence in all our experiments with exploding losses (k) and (k) after only a few epochs.
Algorithms 1 and 2 were implemented using TensorFlow and Keras with GPU acceleration NVIDIA RTX 4090.
C Luuear-quadratic regulator with jumps: detailed results
The value function of the LQR problem with jumps (18) is given by
V(t, x) inf J' ca ||a,1| ds + callx xę
(23)
a
The corresponding HJB equation is
(t, + Tr [BB V
1 2
+ inf
{ailaalB + a VVV((,,))
aERd
+ (A1 + Aa 2 EV x + Z1) v(t,z))
with terminal condition V(T,x) = c2 |01 Using the Ansatz
(t, x) = g(t) + h()))==113
15

---

it is straightforward to see that is of the form
s(t) 1 (TTBBB + Alu) J h(s)ds
(24)
for the unique solution of the non-linear ODE
12(())
(t) 2cg + h(t)uA²
((T) = 202
(25)
(25) can efficiently be solved using a numerical method such as Runge-Kutte of order 5(4); see e.g. Dormand & Prince (1980). The optimal control a is given in terms of h by
h(t)a
(t, 2c1 + h(t)vA2
Note that in the constant intensity case A > 0, = 0, (24)-(25) admit the explicit solutions
g(t) = (Tr[BB"] A1v) c log(
C2 (T - 1) +1)
and h(t) C1 + c2 -)
20102
(26)
C1
But also in the case A₂ 0, numerical solutions of (24)-(25) obtained with Runge Kutta of order 5(4) provide very precise reference solutions V and a to the LQR problem (18).
GPI-PINN and GPI-CBU both need random points (tm, Tm) E [0, T)) x R and (ym) E R2 sampled from two distributions y on [0,T) T) x Ht and on R2, which we chose according to Bachouch et al. (2022).
In the following we consider problem (18) with T B, La 6 = 1, 1/4 and d-dimensional standard normal jumps Zj. Figures 6-8 show results obtained with GPI-CBU in d 50 dimensions for the constant jump intensity case A1 = 1/4 and A₂ = 0, which admits analytical reference solutions.
12.6
V
12.2
Optinal Control e;;
0.2
12.0
0.0
11.8
-0.2
Analytical CPICBU
-0.4
Analytical GPUCBU
1
2
Figure 6: Value function V(0, (left) and first component of the optimal control a (0.x) x) (right) for x
(2), .,....).. 0) with [-2.5, 2.5).. Orange dotted lines: numerical results of GPI-CBU with +1 standard deviation given by orange shaded area. Blue lines: analytical solution (26). 10, A 1/4, Aa = k. 1500.
-0.20
20
Optimal uontrol
Value Function
18
-0.22
36
-0.23
Analytical GPUCBU
Analytical GPL-BU
-0.25
0.4
1.0
10
Figure 7: Value function V(4, x)) (left) and first component of the optimal control aj (1, x) (right) for E [0. 1] and 1op. Orange dotted lines: numerical results from GPI-CBU with 11 standard deviation in orange shaded area. Blue lines: analytical solution (26). d = 50, A1 1/4, A2 = 0, k. = 1500.
16

---

10
008
006 06
02
02
0,20
Figure 8: Heatmaps of MAEv (left) and MÀE. (right) for E [0, 1] and (21, 1,... 1) with
[-2.5, 2.5] obtained from GPI-CBU for LQR problem with 50, A 1/4 and A2 k. 1500.
MAE, in Figure 8 is defined, analogously to MAEv in Section 5.1, by
MAE, = M lagaoo(t;, x;) a (t,, 2;)
a test set of size M uniformly sampled from [0, 1] x [-2.5, 2.5
Next, we consider the same LQR problem (18) as above in 50 dimensions with a controlled jump intensity. Figures 9-10 (and 2), show results of GPI-CBU for A₁ 0 and A2 2. Reference solutions were computed with Runge-Kutte Kutta of order 5(4).
-0.01
Value Function
20
Optinal Control
-0.03
18
-0.04
36
-0.05
14
Reference GPI-CBU
Reference GPLCBU
0.4
0.0
0.2
0.4
0.6
0.8
10
Time
Time
Figure Value function ((t, x) (left) component of optimal control ai x) (right) [0, 1]
and x = 150. Orange dotted lines: numerical results of GPI-CBU with +1 standard deviation in orange shaded area. Blue lines: reference solution. d = 50, A = 0, A2 2. k. 1500.
1.
1.
00
016
00
3
004
004
002
Figure 10: Heatmaps MAEv (left) and MAE, (right) for (0. 1] and = (21, 11) with in
2,5, 2.5] obtained from GPI-CBU for the LQR problem with = 50, A1 = 1/4 and A. = 0. k. 1500.
17

---

These results illustrate the accuracy of GPI-CBU for high-diig -igh-dimnnsional control problems with con- trolled jumps. It can be seen that the optimal control a* lower in magnitude compared to the case of constant jump intensity since now, exerting control increases the likelihood of jumps. But in both cases, optimal to steer the state process X towards as t approaches the time horizon T.
Table summarizes results of GPI-CBU for higher-dimensional LQR problems with the same parameters as above with controlled jump intensity A₁ 0, A2 obtained over h, 5000 training epochs. This demonstrates the scalability of GPI-CBU.
Table 1: GPI-CBU performance metrics as function of the state dimension d for I.QR problems of the form (18) with A1 0, A2 2 and k. 5000 training epochs.
Dimensions d MAEv
MAE,
Loss (0%.)
Loss 22
Time (sec)
0.0023
0.0041
0.0237
-0.1332
6,410
10
0.0025
0.0049
0,0314
-0.1972
8,093
50
0.0147
0.0075
0.1267
-0.4206
16,129
100
0.0492
0.00539
0.6970
-0.4461
24,359
150
0.0979
0.0096
3.7210
-0.4671
33,120
D Optimal consumption-investtmntt problem
D.1 Constant coefficients
We first study the optimal consumptioo-innvestmen problem in model with constant coefficients, where the risk-free asset evolves according to
dsp dtt
(27)
for a constant interest rate r E R, and there are stocks with prices following
M: Σ j=1
ds
pd o' dw; S
4 -1),
(28)
for constant drifts μ¹ E R and volatilities o² R4, n-dimensional Brownian motion W with correlation matrix Σ. independent Poisson processes M with constant intensities X 2 0 and i.id. normal random variables Zi. Z ((zz.o2)) with piz R and oz R4. Suppose an investor starts with an initial endowment of Yo > O, consumes at rate G and invests TiY; i 1,...n. the n stocks. If the risk-free asset used to balance the transactions, the resulting wealth evolves according to
M Σ (e - -1) ==1
dYp Yg
r
Ti(K" -r) -r)- -ce/ dt Σ
T 1=1
o'dw; + d
(29)
Let us assume the investor attempts to maximize
E % 1h (dd
for two utility functions u, U: R. + R and a discount factor E R. Since the driving noise in (29) has stationary and independent increments, it is enough to consider strategies of the form C = c(t, Y and मं = n' (t, Y for functions c, क': (0, x R4 - R,, = 1,..., n. We write this n + 1-dimensional control as = (ce, m!...... .?.....7?)) TT A = R Assuming constant relative risk aversion (CRRA) utility functions u, U with relative risk aversion y > and denoting 5 7 the resulting reward functional is
va (t. v) = J." e -(a--) (c.
ds + xT- (Y
Y =
(30)
8
18

---

Finally, writing P (H² ..'.....'''.. 1²) diag(o' ."" μz -x) oz (o)...., 07) x = (A' ".!...""") - (o'...., T²);, the associated HJB equation for the value function V(4 = sup, (a(t,)) v) satisfies for all (t, E [0, x R.,
{
0 0 (V( (((tv)
- c)
sup
(e,T)En
(31)
T oEo T /² 00 V(, Σ XE (1,,++++ 1)) - vV( + (cy) 2"
with (T) - 1/ /6. This stochastic control problem does not admit an analytical solution but we can instead characterize its solution in terms of PIDE, so to assess the accuracy of the proposed Algorithms 1 and Following Øksendal Sulem (2007), we assume the value function is of the form ytt, A(t)y"/à /8. Then optimizing the HJB equation (31) leads to first order conditions showing that the optimal consumption rate c* is independent from the wealth y and given by
e' (1, = A(1))/1-1))
(32)
and the optimal investment strategy is given by a vector of constants x (t, = x² (m n satisfying >
6-1 =0.
(w-r) + oo T T6 1)+
Σ N
Plugging these optimal controls back into the HJB equation (31), we find that the function A(t) satisfies the ODE
0cA(1) = 0 0- (r+ (u (r) A(1))/6-11) A(t)
1
-06 -1)x To oNaTT
(33)
Σ T ( (1+ -1] - -(1)2/10-11
with A(T) which can be solved numerically using again the Longe-Kutte, method of order 5(4). For both Algorithms and 2, we sample the time and space points independently from the uniform distributions / 0. and 4/ su). respectively. The parameters of the optimal investment problem below
are as follows: T 1, 3b 150, 0.02, 0.045. 5 = 0.7, X = (0.45-1,, fzz 0.25 1,, oz In. Σ 0.2 Inxm with diag (E)
0.2 I,, = 0.032 In,
1, Note that, compared to the LQR problem in Section 5.1, the proportionality factor is now set to 10. Moreover, since A R++1 Dagat is chosen to be the softplus activation function. Instead of the DGM architecture, we here train a classical feedforward neural network with 4 hidden layers, each of 128 neurons. Finally, the MAF and MAF,, metrics are again defined as in Section 5.1 on test set of size M, uniformly sampled from [o, 1] x [0, yo] with V and a obtained from the Runge- Kutta method described above.
In Figure 11 we again compare MAEy and runtime of GPI-PINN and GPI-CBU as functions of 10 stocks in the consumption-investmee
problem (30) with and without jumps. We again see that the assidual-basee GPI-PINN Algorithm tends to be more stable and accurate for larger epochs, despite having higher runtime. When accounting for jumps in the dynamics (29), GPI-PINN becomes numerically very time-consuming even for 10-dimensional control problem. This issue is even amplified compared with the LQR problem (see Figure 1), as the jump size now depends on the control in the wealth dynamics (29). In contrast, GPI-CBU again handles these jumps far more efficiently.
the number of epochs & for
Note that the term (t, y) in (31) is not contained in our general HJB equation (3). But GPI-PINN and GPI-CBU still work if it is added.
19

---

PAN with JUTpS
100
600000
209
Tack
Epech
Comparison of MAEv (bluc) and runtime in seconds (green) of GPI-PINN (solid linc) and GPI- CBU (dashed line) functions of the number of epochs & for the optimal consumption- investment problem without jumps (left) and with jumps (right) n stocks.
Figure 11:
Next, we address stocks), where only GPI-CBU is implemented since GPI-PINN becomes then numerically infeasible. Figures 12 and 13 again confirm the accuracy GPI-CBU for both the value function and the optimal consumption rate ct()), with the standard deviation across 10 independent runs of GPI-CBU being virtually imperceptible. The optimal wealth allocations I²", being constant, are not depicted. However, the MAE., heatmap in Figure 14 corroborates our method's s accuracy in determining both c* and n². also observe that the higher absolute errors tend to occur at the boundaries of the domain.
gigher-dimensional optimal consumption-ing cansumption-invesmeen problem with jumps 50
17
16
Value functonn
15
Velue function
15
14
10
12
Reference
Reference GPUCBU
80
100
120
140
0.0
0.2
0.4
0.6
0.8
10
E [0. 150] (left) and E 1]. y 50 (right) for the optimal consumption-invesmeen problem (20) with constant coefficients and n = 50 stocks. Orange dotted lines: results of GPI-CBU with + standard deviation given by orange shaded area (k, = 1000). Blue lines: reference solution from Runge-Kutta applied to (33).
Figure 12: Value function (tt, y) for
10
0.9
Optimal consumption
0.6
Reference GPI-CBU
05
0.4
0.8
10
Time
Figure 13: Optimal relative consumption rate (t) for t E [0, for the optimal consumption consumption-investueee investment problem (20) with constant coefficients and 50 stocks. Orange dotted line: results of GPI-CBU with + standard deviation given by orange shaded area (k. 1000). Blue line: reference solution from Runge- Kutta applicd to (32)-333).
20

---

10
0420
Tine
002
001
80
200
Weathy
Figure 14: Heatmaps of MAEv (left) and MAE, (right) for t E [0, 1] and y E [o, 150). for GPI-CBU applied to the optimal consumpiion-investenn problem (20) with constant coefficients and 50 stocks.
D.2 Stochastic coefficients
Now, we study more complex optimal consumptiooninnvesitnnn problem in realistic market of the form (27)- with a stochastic interest rate
dr; = (b; - r2 rz) + Ur V dWT,
with stochastic (Heston) volatility models for the n stock prices
M <
dS S
dt + of dW; +
( (1)
1=1
for of u/ where of follows
de - at - uz) + vs dw...
(34)
and with stochastic jump intensities
dx = (b/ x + V Af awa..
We denote W (WS.. S.... We.1 ..... W W W WT)T (3n + 1)-
dimensional Brownian motion with correlation matrix E. In this case, the wealth process can still be described follows
dY Y2
V
Mi ( ((6i-1) 2 1=1
rr
+
Σ 1=1
Tilu" rz) dl +
and the strategies should be of the form c(t, Ut, de, Tt, r2) and (t, vt, Ae, rr, Y.)., with at (ct, ......,77 ,,," TE A == Rp+1 Hence, the dynamics of the (2n + 2) dimenssionll process X
(ve, Ne, Y are given by
M
dxę = (t, Xt X.)
7x X
c -1)
with px(+) E 22=+2 Ex(-) E R (2
and 7x E 2m2 such that
a (b. vz)
O.
ax(bx Aa)
O,
Xi)
xx(t,
ar (b ri)
y (re + (μ - re) Te c.)
ya क. 1,
21

---

and
Onxn
Dv
Onxn
Onx1
Onxn
Onxn
Dx
0nx1
Ex(t,
01xn
01 01xn
01xn
Vrv
Dy
01 x
01xn
where D. = diag 1/ Vevee... //...... 1 7
D diag
11
17
Dy =
Yy Vot ................................ Y oo 1xn
Therefore, the value function
e p(a- (o) ds t P(T (v
x)t sup E
X =
6
a
satisfies the following HJB equation for all (4,x) E (0, x R x R4 R x R++
0,V(t,z) - -V((t,)
sup (cx))
V+V(t,) | Σx (t, EX) (tv,,
v(t,x)) v(t,)) + (cy)
Σ A Vitu,
(35)
with terminal condition xT) /5. We consider portfolio of = 25 stocks, resulting in 52- diddessiionll value function and a 26-dimensional control process, with the same parameters as in Section D.1. This version of the consumption inveettennn problem is significantly more complex than the one discussed in Section D.I, with the value function's 's dimensionality increasing from 2 to 52. Consequently, Runge-Kutta can longer be used solve the associated HJB equation (35), making it impossible to compute the MAE and MA E.z metrics. However, despite this lack of a reference for comparison, GPI-CBU still produces results that appear reasonable. We indeed first observe in
Figure 4 (main manuscript) that the losses _2 (14) and 2 (16) converge as the number of epoch l increases. Note that the orange curve in the left plot represents the interior loss (first right-hand see Eq.
term) of k)k while the blue curve is the boundary loss (second right-hand term) of k (14).
The following Figures 15 and 16 confirm that the results of GPI-CBU for both the value function and optimal control are reasonable. as they closely resemble those of Figures 12 and 13. For the value functions of Figure 15, the standard deviations across 10 independent runs of GPI-CBU remain very low, confirming the stability of the approximations For the optimal consumption rate in Figure 16 (left plot), the standard deviation across the 10 runs tends be higher for values of time close to although being still reasonable. Finally, varying the first dimension of the intensity X mainly impacts the corresponding fraction of wealth T. while its effect on the other proportions and consumption rate is more moderate (since arising from the correlation matrix Σ of the Brownian motion W).
17
16
functon
15
Value function
15
10
13
5-
12
80
100
120
140
0.0
0.2
0.4
0.6
0.8
100
Weath
Figure 15: Value function (tt, ()) for & and x (0.15 1 110, 0.02, 3) with y [0, 150] (left) and for t [0, 1] and (0.15- L10, 110, 0.02, 50) (right), for optimal consumption-investtteeeeeteeetee investment problem (20) with stochastic coefficients and n 25 stocks. Blue lines: results of GPI-CBU with I standard deviation given by orange shaded area (k. = 1000).
22

---

10
15
Optimal consumption rate
Optimal control a
0.8
0.
0.6
0.0
0.2
0.4 0.6
0.8
1.0
Intersity
Figure 16: Optimal policy for the optimal consumption -ineinittiooo-nnemmetttttt problem (20) with stochastic coefficients and
= 25 stocks. Left plot: optimal consumption rate c² (f, 2) for t E [0, 1] and x = (0.15 - 110, 1 0.02, 50), results of GPI-CBU in blue and = 1 standard deviation given by orange shaded area (k, = 1000). Right plot: optimal consumption rate (f) and first five optimal fractions of wealth x' (t, for 0 and (0.15. 10; A 19, 0.02, 50) for X [0, 5].
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Every claim in the abstract and introduction is either theoretically numeri- cally supported in the paper. Similarly, the Conclusion, limitations and future work Sections discuss the limitations and possible extensions of the current work.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: The main limitation of our approach lies in the need to know the underlying dynamics of the state process, which not always available in real-world applications. While it is reasonable to assume that physical systems obey known laws of motion, the dynamics in economies and finance often need be estimated. Moreover, theoretical convergence guarantees for the GPI-PINN and GPI-CBU still need to be established in future research. This has been emphasized as a separate part of the conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: In the novel Propositions 3.1 and 4.1, all assumptions required to establish the results are clearly stated (or corresponding references provided). Similarly, for the well-known (standard) Theorem 2.2.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Sections 3-4 describes how to implement the two algorithms proposed in this paper with detailed pscudo-code. Sufficient information are also disclosed in Section 5 and Appendix for reproducing the associated numerical results. The code will be made open access upon acceptance of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
25

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code is available http::///ggttaii.como////rereeep
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnps.cc cc/
casssoonPlccyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so No an acceptable answer. Papers canmot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: /nips. /nips.cc/ppui wtt.cc/cccccccccccccceoooo /owpdddeeeeoodeeeiiiiiiiiiieeeee soooolioiii for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The training and testig details are specified in Section 5 and the Appendix B. Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All relevant figures presented with + +1-sigma confidence intervals. This is stated every time needed and the computation of these standard errors is explained in Section 5.
Guidelines:
26

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please consult the supplementary material in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https://neurii: cttpe////ccciciicceeeetttttt weneeuideeeineeeee ines?
Answer: [Yes]
Justification:
The research conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction)
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
27

---

This paper presents work whose goal is to advance the field of Machine Learning by proposing new deep-learning upproach for solving stochastic optimal control problems. It has therefore no direct societal impact.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-uss should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification:
We do not use any existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
28

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi chcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release for now new assets. The related code will be made open access upon acceptance of the paper.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve any crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
supplemental material is fine, but if the main contribu-
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
29

---

The paper does not involve any crowdsourcing nor research with human subjects.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: This research does not involve LLMs as any important, original, or non- standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: hatpps:///eurps..ooo http://rrrp.s.ccoooorer///////// //tp////eeeeeeeeee///////////// for what should or should not be described.
30


Reviews:
[{"review_id": "XVdyUi60OI", "reviewer": "Reviewer_fySF", "summary": "This paper introduces a model-based deep learning approach for solving finite-horizon continuous-time stochastic control problems that feature jumps. The proposed method iteratively trains two neural networks: one to represent the optimal policy and another to approximate the value function. Leveraging a continuous-time version of the dynamic programming principle, the authors derive two distinct training objectives based on the Hamilton-Jacobi-Bellman (HJB) equation, which aim to ensure the networks capture the underlying stochastic dynamics. They propose two algorithms, GPI-PINN 1 and GPI-PINN 2. GPI-PINN 1 minimizes HJB residuals, but is noted to be inefficient with jumps. GPI-PINN 2 is designed to circumvent costly computations of gradients, Hessians, and jump expectations, making it efficient for high-dimensional problems with jumps. Empirical evaluations on various problems, including linear-quadratic regulators and optimal consumption-investment problems (both with and without jumps), illustrate the accuracy and scalability of their approach, demonstrating its effectiveness in solving complex, high-dimensional stochastic control tasks.", "questions": "1.  In Algorithm 2, the function $\\mathcal{G}$ plays a crucial role in updating the value and control networks. Could the authors provide a more intuitive explanation for how $\\mathcal{G}$ relates to the original HJB equation and why its use helps circumvent the costly computations of expectations and higher-order derivatives? A deeper dive into the connection between minimizing $\\mathcal{G}$ and satisfying the optimality conditions would enhance clarity.\n2.  The paper highlights that model-free RL is less accurate when dynamics are known. Could the authors elaborate on how their approach (which relies on *known* dynamics) compares to recent advancements in model-learning within RL, where models are learned from data and then used for planning or control? This would provide a broader context for the \"model-based\" claim.\n3. The paper mentions various hyperparameters (e.g., $M_1, M_2, k_*, \\eta_1, \\eta_2$, $\\xi$, network architecture). A more detailed discussion on the sensitivity of the algorithms (especially GPI-PINN 2) to these parameters, and strategies for their robust selection across different problem instances, would be valuable for practical implementation. What are the criteria for selecting the proportionality factor $\\xi$ (L209) and its impact on convergence? \n4. The current formulation assumes specific characteristics for the random measure $N^\\alpha$ and the distribution $Z$ of jump vectors. How general is the approach for different types of jump processes (e.g., non-Poissonian jumps, jumps with state-dependent or control-dependent distributions that are more complex than the assumed form)?\n5.  While GPI-PINN 2 is stated to be more efficient, a more explicit computational complexity analysis (e.g., in terms of state dimension $d$, number of hidden layers $L$, and samples $M_1, M_2$) for both training and inference, compared to alternative methods, would solidify the claims about scalability.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "This paper presents a rigorous and detailed model-based deep learning approach to a challenging class of stochastic control problems. Its strengths lie in its theoretical foundation and the proposed methods' efficiency for high-dimensional problems with jumps. However, there are some areas where the clarity and comprehensiveness of the presentation could be improved.\n\n**Strengths:**\n*  The paper tackles continuous-time stochastic control problems with jumps in high dimensions, which is a complex and relevant area. The development of GPI-PINN 2, specifically designed to handle jumps efficiently by avoiding computationally expensive gradient, Hessian, and expectation computations, is a notable contribution.\n* Unlike many model-free deep RL algorithms that rely solely on environmental sampling, this approach explicitly leverages known underlying dynamics (HJB equation). This design choice is claimed to lead to higher accuracy when state dynamics are fully known.\n* The iterative training of two neural networks for both the value function and optimal control provides a global approximation across the entire space-time domain, allowing for rapid online evaluation. This is an advantage over local approaches.\n*  The method's effectiveness, scalability, and accuracy are demonstrated across various numerical examples. These include LQR problems with and without controlled jump intensities, and complex optimal consumption-investment problems.\n* The approach is well-grounded in stochastic control theory, leveraging the HJB equation and providing propositions (2.2 and 2.3) that enable computational efficiencies.\n\n**Weaknesses:**\n* While propositions are provided, the transition from theoretical statements to the practical implementation within the algorithms (e.g., the exact form of $\\mathcal{G}$ in Algorithm 2's update rules) could be clearer in the main text. Some key definitions from standard literature are referenced but full clarity requires consulting external sources.\n* While the paper states that model-free RL algorithms are \"less accurate in cases where the state dynamics (2) are fully known and given\" and shows GPI-PINN 2 outperforming PPO and SAC in Figure 3, the explanation for why this is the case could be more deeply explored beyond \"rely solely on sampling\". A more nuanced discussion of the trade-offs (e.g., data efficiency, robustness to misspecification) would be beneficial.\n* The paper identifies that a limitation is the need to know the underlying dynamics of the state process, which is \"not always available in real-world applications\". While this is acknowledged, the implications for practical deployment are significant and warrant further discussion on how this model-based approach would integrate with real-world scenarios where dynamics need to be learned or estimated.\n* GPI-PINN 1's inefficiency with jumps is a major drawback. While GPI-PINN 2 addresses this, the practical utility of GPI-PINN 1 seems limited, and perhaps it should be framed more as a stepping stone to GPI-PINN 2 rather than a fully viable alternative.", "was_revised": true, "final_justification": "Thanks for the clarification. However, I don't see this paper with \"high impact on at least one sub-area of AI or moderate-to-high impact on more than one area of AI\", so I will maintain my score to borderline accept this paper.", "rebuttal": "**Q1:** *In Algorithm 2, the function $\\mathcal{G}$  plays a crucial role in updating the value and control networks. Could the authors provide a more intuitive explanation for how  $\\mathcal{G}$  relates to the original HJB equation and why its use helps circumvent the costly computations of expectations and higher-order derivatives? A deeper dive into the connection between minimizing $\\mathcal{G}$  and satisfying the optimality conditions would enhance clarity.*\n\n**Answer:** Thanks for asking. We will try to explain this better. A typical Hamiltonian contains the gradient and the Hessian of the value function. This becomes computationally prohibitive in high dimensions (say 20 or more). Moreover, if the control problem contains jumps, the Hamiltonian includes jump-expectations, which in general, have to be computed with nested Monte Carlo. This increases the computational effort further.\n\n$\\mathcal{G}$ is an expectation-free version of the Hamiltonian, which bypasses the computation of the Hessian and jump-expectations and which can be interpreted as the expectation-free HJB residuals. Proposition 2.3 then uses the $L^2$-optimality property of conditional expectations to show that the value function $V^\\alpha$ can be learned from $\\mathcal{G}$ using the update rule (12) . \n\nMoreover, the GPI-PINN 2 update rule only requires the computation of $\\nabla_\\theta V_\\theta(t,x)$, eliminating the need for backpropagation through the expectation-free Hamiltonian $\\mathcal{G}$, which is evaluated using the previous weights $\\theta^{(k-1)}$. While GPI-PINN 1 gives good results for low-dimensional problems without jumps, GPI-PINN 2 is much more efficient for high-dimensional problems with jumps; see also our answer to Q1 of Reviewer 4.\n\n**Q2:** *The paper highlights that model-free RL is less accurate when dynamics are known. Could the authors elaborate on how their approach (which relies on known dynamics) compares to recent advancements in model-learning within RL, where models are learned from data and then used for planning or control? This would provide a broader context for the \"model-based\" claim.*\n\n**Answer:** Thanks, this is a good remark. Our approach indeed requires that the dynamics of the underlying system are known. But nothing prevents us from learning these dynamics in a preliminary step from data using a recent model-learning (RL) algorithm such as Brunton et al. (2016) or Champion et al. (2019). Once an approximate model has been learned, GPI-PINN  can then be applied to efficiently solve the resulting control problem. Including this perspective in our paper is a good idea. It will provide valuable context and broaden its scope. \n\n**Q3:** The paper mentions various hyperparameters (e.g., $M_1, M_2, \\eta_1, \\eta_2, \\xi$, network architecture). A more detailed discussion on the sensitivity of the algorithms (especially GPI-PINN 2) to these parameters, and strategies for their robust selection across different problem instances, would be valuable for practical implementation. What are the criteria for selecting the proportionality factor*  $\\xi$ (L.209) and its impact on convergence?\n\n**Answer:** Yes, you are right. Figures 1 and 11 show the sensitivity of our algorithms to the number of epochs $k^*$. Additional results should also be included for our grid search over $M_1, M_2, \\eta_1, \\eta_2$, although we observed that our algorithms' performance is robust with respect to these parameters (for $M_1, M_2$ sufficiently large). The most crucial hyperparameter for GPI-PINNs convergence is $\\xi$, which we determined with the approach of Wang et al. (2022). They have shown that in the approximations of solutions of PDEs with infinite-width network (in the limit), the convergence rate of the PINN training error depends on the singular values of the neural tangent kernel associated with the PINN (which they derive explicitly). By choosing the proportionality parameters $\\xi$ to be proportional to these singular values, they demonstrate empirically that the numerical convergence of PINNs and related algorithms can be significantly improved. We will emphasize this more clearly in Appendix 6.3, where the neural network architecture is described.\n\n**Q4:** *The current formulation assumes specific characteristics for the random measure  $N^\\alpha$ and the distribution $Z$ of jump vectors. How general is the approach for different types of jump processes (e.g., non-Poissonian jumps, jumps with state-dependent or control-dependent distributions that are more complex than the assumed form)?*\n\n**Answer:** Our framework is very general. It covers non-Poissonian jump arrivals. Both, the jump intensities and jump-size distributions can depend on the time, state and control. The only thing it does not cover is infinite jump actvity, i.e. infinitely many small jumps during compact time-intervals. But such processes are a theoretical construct, which numerically, have to be approximated, typically by ignoring all jumps below a given threshold.\n\n**Q5:** *While GPI-PINN 2 is stated to be more efficient, a more explicit computational complexity analysis (e.g., in terms of state dimension $d$, number of hidden layers $L$, and samples $M_1$, $M_2$) for both training and inference, compared to alternative methods, would solidify the claims about scalability.*\n\n**Answer:** Thank you for the suggestion. A complete theoretical analysis is difficult. But we have some partial answers, and our numerical experiments strongly suggest that GPI-PINN 2 scales well. \n\nAt inference, the computational complexity for evaluating the value network $V_\\theta$ with depth $L$ and width $W$ is the same for GPI-PINN 1 and 2, namely \n$$\\mathcal{O}(M_1 LW^2).$$\n\nRegarding training time, we have for GPI-PINN 1, \n$$\\mathcal{O}(M_1 M_3 d^2LW^2+ M_2 LW^2) = \\mathcal{O}(M_1 M_3 d^2LW^2),$$\n\nand for GPI-PINN 2,\n$$\\mathcal{O}(M_1 dLW^2 + M_2 LW^2) = \\mathcal{O}(M_1 dLW^2),$$\nwhere $M_3$ is the number of samples used to approximate the jump expectation $\\mathbb{E}^{\\mathcal{Z}}$ in GPI-PINN 1. This highlights the significant reduction in computational cost achieved by GPI-PINN 2 for training the value network $V_\\theta$ in high-dimensional control problems with jumps, as confirmed by Figure 1. For the control network  $\\alpha_\\phi$, the training phase (Step 2) is the same between the two algorithms. Additional comparisons with existing deep learning algorithms could also be provided.\n\n- Brunton, S. L., Proctor, J. L., & Kutz, J. N. (2016). Discovering governing equations from data by sparse identification of nonlinear dynamical systems. _Proceedings of the national academy of sciences_, _113_(15), 3932-3937.\n- Champion, K., Lusch, B., Kutz, J. N., & Brunton, S. L. (2019). Data-driven discovery of coordinates and governing equations. _Proceedings of the National Academy of Sciences_, _116_(45), 22445-22451.\n- Wang, S., Yu, X., & Perdikaris, P. (2022). When and why PINNs fail to train: A neural tangent kernel perspective. _Journal of Computational Physics_, _449_, 110768."}, {"review_id": "8kCrydAwUr", "reviewer": "Reviewer_cXcY", "summary": "This paper investigates a finite-horizon continuous-time stochastic control problem with jumps. The associated Hamilton–Jacobi–Bellman (HJB) equation includes a nonlocal integral term due to the jumps, which makes the problem harder than normal optimal control problem. The authors propose a two-step iterative scheme: for a fixed feedback control, the value function is updated by minimizing the residual of the HJB equation; then, holding the value function fixed, the control is improved by minimizing the Hamiltonian. Two variants of the algorithm are introduced: the first uses multiple samples to approximate the jump expectation via Monte Carlo, while the second simplifies this step by using a single sample per spatial point. Numerical experiments are presented to demonstrate the performance and efficiency of both approaches. Overall, the paper is well-written.", "questions": "The design of Algorithm 2 raises two conceptual concerns that should be clarified. I would suggest the paper be accepted if the authors could address this issue.\n\nIn the first part, you modified the PINN loss by fixing the parameter $\\theta$ in G and only optimize the term $V_\\theta$ in line 191.5. (I temporally ignore the boundary trem.) Reorganizing the terms, the bracket before taking square in loss $\\mathcal{L}_1$ can be viewed as\n\n$$V_\\theta – V_{\\theta^{(k)}} – RES_{HJB}(\\theta^{(k)}),$$\n\nwhere $RES_{HJB}$ is the residual of the HJB equation (before taking expectation) in the PINN loss. This update means we hope that the new value function is close to the old one added by the HJB residual. However, since $E[RES_{HJB}] = 0$ characterizes the true solution, there's no justification for using the residual itself as an additive update. In fact, using\n\n$$V_\\theta – V_{\\theta^{(k)}} + RES_{HJB}(\\theta^{(k)}),$$\n\n(the negative of the update) seems equally justified, leading to opposite update directions. The current construction appears arbitrary without additional justification.\n\nFor the second part, the Hamiltonian is expected to be maximized after taking expectation, while the algorithm only uses one sample for each x. Please explain why it is fine we only use one sample?\n\nBelow are some minor questions or comments.\n\n1.\tPage 2 line 71. Duan et al is not the original paper for TRPO. It is better to cite the original work.\n\n2.\tPage 3 line 104. HBJ should be HJB.\n\n3.\tPage 4 eq (11). One the left, should it be just a value function without the input?\n\n4.\tPage 5, part 2 of the first algorithm. I would suggest the authors add a remark that the term $\\partial_t v$ in $\\mathcal{H}$ does not involve I the optimization, so it is just minimizing the Hamiltonian. (At first glance, it seems you are updating the control using the residual of the HJB equation.)\n\n5.\tThe algorithm that consecutively update the value function and the control is very similar to the actor-critic methos in reinforcement learning. I think papers that implement this method in optimal control method such as\nSolving time-continuous stochastic optimal control problems: Algorithm design and convergence analysis of actor-critic flow\nCan be added to related work.", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n\nThe paper is well-written, with key concepts and algorithmic steps clearly explained.\n\nThe use of Proposition 2.2 enables an efficient approximation of the second-order differential operator without computing gradients and Hessians directly, which is beneficial for high-dimensional problems.\n\nThe second algorithm reduce computational costs.\n\nThe authors present detailed numerical experiments\n\nWeakness\n\nWhile the second algorithm reduces the computational cost, it lacks theoretical justification. I have put details in Questions.", "was_revised": true, "final_justification": "The authors resolved most of my questions. My recommendation is not changed.", "rebuttal": "**Q1:** *In the first part, you modified the PINN loss by fixing the parameter  $\\theta$ in $G$ and only optimize the term  $V_\\theta$  in line 191.5. (I temporally ignore the boundary trem.) Reorganizing the terms, the bracket before taking square in loss  $\\mathcal{L_1}$ can be viewed as*\n$$V_\\theta - V_{\\theta_k} - RES_{HJB}(\\theta^{(k)})$$ \n\n*where $RES_{HJB}$  is the residual of the HJB equation (before taking expectation) in the PINN loss. This update means we hope that the new value function is close to the old one added by the HJB residual. However, since  $\\mathbb{E}[RES_{HJB}]$  characterizes the true solution, there's no justification for using the residual itself as an additive update. In fact, using*\n$$V_\\theta - V_{\\theta_k} + RES_{HJB}(\\theta^{(k)})$$\n\n*(the negative of the update) seems equally justified, leading to opposite update directions. The current construction appears arbitrary without additional justification.*\n\n  \n**Answer:** This is an excellent remark. More generally, using Proposition 2.3, we can always rewrite the bracket in $\\mathcal{L}^1$ (l.191) as\n$$V_\\theta - V_{\\theta_k} - \\pi_k RES_{HJB}(\\theta^{(k)}) \\ ,$$\n\nwhere $\\pi_k \\in \\mathbb{R}$ is an adaptive scaling factor for the expectation-free HJB residuals that may depend on the epoch $k$. \n\n\nFrom a practical perspective, we empirically observe that the simple rule $\\pi_k \\approx 1$ provides a good trade-off between convergence speed and accuracy of the improvements in GPI-PINN 2. Exponentially decaying  $\\pi_k > 0$ also perform well in practice (similarly to learning rate schedules), while we observe that negative scaling factors always fail to converge to the true solutions with exploding losses $\\mathcal{L}^1$ and $\\mathcal{L}^2$ after only a few epochs.\n\n\nTheoretically, this is related to the convergence properties of the GPI-PINN 2 algorithm, which are still under research. But roughly, the observed empirical behavior can be understood by considering the following extended operator at epoch $k$,\n$${\\small\\mathcal{T}^k V \\hspace{-0.5mm}= \\hspace{-0.5mm} V  \\hspace{-0.5mm}+ \\pi_k  \\hspace{-0.5mm}\\left[ RES_{HJB}(V)+\\frac{h}{2} RES^2_{HJB}(V)+ \\frac{h^2}{3} RES^3_{HJB}(V) + \\cdots \\right]} , $$ \n\nsuch that $\\theta^{(k+1)} = \\mathcal{T}^k V_{\\theta^{(k)}}$. This extended operator $\\mathcal{T}^k$ can be shown to be contractive for specific values of $\\pi_k > h >0$, while it is always expanding for $\\pi_k <0$. For simplicity, we did not include this extension in the original manuscript, but we agree that incorporating these empirical observations along with detailed numerical results and a brief theoretical justification concerning the choice of $\\pi_k$, could strengthen a revised version of the manuscript.\n\n**Q2:** *For the second part, the Hamiltonian is expected to be maximized after taking expectation, while the algorithm only uses one sample for each x. Please explain why it is fine we only use one sample?*\n\n**Answer:** Yes, correct. This is a bit like stochastic gradient descent. Every single sample point involves an error. But over many iterations, the errors average out. We see in our numerical experiments that we have more oscillations in the training of GPI-PINN 2 compared to GPI-PINN 1, but the loss still goes to zero. See Duarte et al. (2024), Duan et al. (2023) and Cohen et al. (2025) for theoretical convergence results supporting this approach. \n\n**Q3:** *Below are some minor questions or comments.*\n\n1.  *Page 2 line 71. Duan et al is not the original paper for TRPO. It is better to cite the original work. Page 3 line 104. HBJ should be HJB.*\n\nYes, thank you. We will refer to the original paper.\n    \n2.  *Page 4 eq (11). One the left, should it be just a value function without the input?*\n\nThanks, but we need the inputs since (11) describes a non-linear regression of $\\mathcal{G}$ on $X^\\alpha_t$ at time $t$.\n    \n3.  *Page 5, part 2 of the first algorithm. I would suggest the authors add a remark that the term  $\\partial_t v$  in  $H$  does not involve in the optimization, so it is just minimizing the Hamiltonian. (At first glance, it seems you are updating the control using the residual of the HJB equation.)*\n\nIndeed, you are right.\n    \n5.  *The algorithm that consecutively update the value function and the control is very similar to the actor-critic methos in reinforcement learning. I think papers that implement this method in optimal control method such as Solving time-continuous stochastic optimal control problems: Algorithm design and convergence analysis of actor-critic flow Can be added to related work.*\n\nYes, you are right. We will add these references.\n\nThank you for the reference, this will be added. \n\n\n- Cohen, S. N., Hebner, J., Jiang, D., & Sirignano, J. (2025). Neural Actor-Critic Methods for Hamilton-Jacobi-Bellman PDEs: Asymptotic Analysis and Numerical Studies. _arXiv preprint arXiv:2507.06428_. \n- Duan, J., Li, J., Ge, Q., Li, S. E., Bujarbaruah, M., Ma, F., & Zhang, D. (2023). Relaxed actor-critic with convergence guarantees for continuous-time optimal control of nonlinear systems. _IEEE Transactions on Intelligent Vehicles_, _8_(5), 3299-3311. \n- Duarte, V., Duarte, D., & Silva, D. H. (2024). Machine learning for continuous-time finance. _The Review of Financial Studies_, _37_(11), 3217-3271."}, {"review_id": "7cb9mFwxNa", "reviewer": "Reviewer_iDp5", "summary": "In this paper, past works in deep learning for continuous-time stochastic control are extended by utilizing physics-informed neural networks to reduce the computational burden of learning the optimal control and value function through generalized policy iteration. The first algorithm introduced, GPI-PINN-1, trains the value network to satisfy the Hamilton-Jacobi-Bellman equation by setting the loss function to be the residual. In this version the authors bypass computing the gradients and Hessian of the value function, but still need to evaluate the jump expectations in order to evaluate the Hamiltonian. GPI-PINN-2 not only bypasses the gradient computations of all orders, but also avoids explicitly evaluating the jump expectations by exploiting the known state dynamics, since the value target can be expressed in terms of these quantities. Furthermore, they use an iterative approach where the value network learned in the past iteration step is involved in computing the target for the new iteration step. However, this computational efficiency comes at the cost of reduced stability during training. The performance and training time of GPI-PINN-1 and 2 are compared, revealing that GPI-PINN-2 is by far the more practical algorithm when jumps must be accounted for despite its stability issues. Finally, numerical experiments on continuous-time stochastic control demonstrate the superiority of GPI-PINN-2 over existing model-free and model-based algorithms.", "questions": "**Q1:** How is the stopping condition in algorithm 1 evaluated efficiently?\n\n**Q2:** How high-dimensional can the problem get before GPI-PINN-2 struggles to solve it?\n\n**Q3:** What was the actual batch size per forward pass used when training the GPI-PINN networks? Was this batch size (and the number of batches per epoch) also used for the baseline deep learning algorithms?", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Quality:** The techniques used in this paper have a solid theoretical foundation, and this is reflected in the strong empirical performance of GPI-PINN-1 and 2. The experiments run give a good idea of the performance benefits that GPI-PINN-2 can provide over GPI-PINN-1 and the baseline methods, validating that it would be worth adopting is many scenarios. Overall, I consider this to be high-quality work.\n\n**Clarity:** The overall clarity of the paper is good, with the algorithms being presented cleanly and the figures mostly being easy to interpret. However, there is a mismatch between the legend of the left plot of Figure 4 (which says the blue curve is the boundary loss) and the caption (which says the orange curve is the boundary loss). Furthermore, with the large amount of notation which needs to be introduced in the first two sections, adding a table in the appendix explaining each symbol may be a good idea.\n\nThere are a few questions which I had about the practical implementation and performance of the algorithms, which are given in the questions section.\n\n**Significance:** The problem formulation is very general and could be applied to many control problems, with the caveat that the system dynamics must be known. As the authors state, there are plenty of problems where certain dynamics can be modeled using known physical laws; however, in a real-life system the overall dynamics are likely to be influenced by less predictable phenomena as well. Furthermore, it is not obvious that a model trained in this way will be robust to noisy state measurements during deployment.\n\n**Originality:** This is the first application of physics-informed neural networks to this problem that I am aware of. However, the methods used are closely related to those introduced in [1], including the use of the least-squares Monte-Carlo method to bypass computing the jump expectation.\n\n[1] Duarte, V., Duarte, D., and Silva, D. H. Machine learning for continuous-time finance. The Review of Financial Studies, 37(11):3217–3271, 2024.", "was_revised": true, "final_justification": "Overall, I believe that the authors have responded to the concerns of myself and the other reviewers effectively. Hence, I am raising my score to a 5.\n\nIssues resolved: \n- An explanation of how the stopping condition is evaluated efficiently (using samples that provide coverage of the entire state space and all possible points in time) was given. If the maximum error over all sample points remains small over several rounds, it is reasonable to assume that the value network has learned a good approximation of the value over the entire state space.\n- The authors are prepared to show experimental results backing up the claim that GPI-PINN-2 can handle problems with more dimensions than were already considered in the paper\n- The authors confirmed that the comparison between their method and the baselines was fair in terms of the time used for training\n- The authors have shown that they can provide additional theoretical justifications for their choices in their responses to reviewers fySF and cXcY\n\nFurthermore, I do not consider the similarities between this paper and the paper by Duarte to be grounds for rejection.", "rebuttal": "**Q1:**  *How is the stopping condition in algorithm 1 evaluated efficiently?*\n\n**Answer:** Thank you, we should have explained this more clearly. Since our value function is parameterized by a neural network $V_\\theta$, it is straightforward to evaluate the difference between $V_{\\theta^{(k)}}(t,x)$ and $V_{\\theta^{(k-1)}}(t,x)$ on a set of sample points $(t,x)$ from $[0,T] \\times D$. In practice, we repeat *Step 1* (value updating) $N_1$ times and *Step 2* (policy updating) $N_2$ times before evaluating the stopping condition. An alternative efficient stopping rule would be to simply fix a maximum number of epochs $k^*$ (as used in the comparison Figure 3). This should indeed be emphasized more clearly. A more implementable version of GPI-PINN 1 will also be added to the Appendix for clarity.\n\n**Q2:**  *How high-dimensional can the problem get before GPI-PINN-2 struggles to solve it?*\n\n**Answer:** In all our numerical experiments, GPI-PINN 2 can easily handle control problems with up to 100-150 dimensions. On the other hand, in our comparison study, GPI-PINN 1 cannot scale beyond 10 dimensions with jumps and 50 dimensions without jumps, which confirms the efficiency of GPI-PINN 2 as outlined in Figure 1 (see also the computational complexity analysis provided to Reviewer 1, Q5). For this reason, the original manuscript only reported results for both GPI-PINNs up to 50 dimensions.  However, higher-dimensional results for GPI-PINN 2 are ready and can of course be included to further confirm the scalability of the approach.\n\n**Q3:**  *What was the actual batch size per forward pass used when training the GPI-PINN networks? Was this batch size (and the number of batches per epoch) also used for the baseline deep learning algorithms?*\n\n**Answer:** We used a batch size of 256 with a number of steps $N_1 = N_2 =64$ when training the GPI-PINNs, leading to a total batch size of 16,384 per epoch $k$ before each update.  The batch size in the other baseline algorithms was configured in such a way to ensure roughly equal computing time per iteration in each method, so that a similar comparison plot as Figure 3 would be obtained if we replaced the epochs by the actual running time in seconds (*i.e.* batch size of 512 with approximately 64 steps per epoch for PPO, SAC, Han and E). We will emphasize this more clearly in Appendix 6.3. when describing the neural network architecture.\n\n**Originality:** Our method is indeed related to the paper \"Machine learning for continuous-time finance (2024)\" by Duarte, Duarte and Silva. Their main focus is on lower-dimensional problems without jumps. But they also solve a 1-dimensional example with jumps. Our value function update rule is different, and we consider high-dimensional problems with finite time horizon. Since we have a finite time horizon, we also have a terminal condition, which we write as an additional term into the loss function."}, {"review_id": "Hoxx7IFaDt", "reviewer": "Reviewer_9fvu", "summary": "This paper introduces a deep learning approach to solve stochastic optimal control problems with jumps. The method can be interpreted as a PINN-style approach to solving the HJB of the optimal control problem. Since the HJB is second order, a PINN style approach would require computing third-order derivatives, which quickly becomes infeasible. The paper proposes a second algorithm that circumvents this issue using a recursive approach instead of relying solely on HJB residuals. Theoretical results are accompanied with numerical results on small dimension settings.", "questions": "1. Can the authors provide a condensed summary of differences between algorithm 1 and 2?\n2. The experiments for consumption-investment do not compare the proposed algorithm with other standard DL frameworks as done for the LQR setting. Could the authors share these experimental results as well?\n3. Could the authors comment on the scalability and potential of this approach to larger scale environments? Are there environments or settings that are particularly suited to the strengths of the proposed algorithms?", "limitations": "Yes, the authors have adequately addressed the limitations of their work.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Theoretical results are sound and proofs seem correct.\n2. The main algorithms provided are original and introduce a new approach to solving optimal control problems.\n3. The writing is thorough and comprehensive, and the mathematical results are precise.\n\nWeaknesses:\n1. Some of the exposition and notation is extremely dense and hard to follow.\n2. Numerical experiments seem to only address dimensions d=10 to 50 and are limited to LQR and consumption-investment problem. It would be nice to mention larger scale problems where this could be applied.\n3. The quality of the plots is unclear and makes it hard to read in some cases.", "was_revised": true, "final_justification": "While the authors did share improved results for slightly higher dimensionality, certain concerns from other reviewers regarding theoretical concerns make me unsure of raising my score, hence I shall maintain my score of 4.", "rebuttal": "**Q1:**  *Can the authors provide a condensed summary of differences between algorithm 1 and 2?*\n\n**Answer:** Sure, both are actor-critic methods using two neural networks $V_\\theta$ and $\\alpha_\\phi$ to approximate the value function and optimal control, respectively. \n\nGPI-PINN 1 uses a PINN approach to approximate the value function by minimizing the residuals $\\mathcal{H}$ of the HJB equation (4). It works well in low dimensions and without jumps but becomes inefficient otherwise since it needs to compute the Hessian and the jump-expectations $\\mathbb{E}^{\\mathcal{Z}} V_\\theta(t, x + \\gamma(t,x,Z_1,\\alpha_\\phi(t,x))$ in $\\mathcal{H}$ for each sample $(t,x)$ in every iteration. Additionally, it requires the backpropagation through the second-order Hamiltonian $\\nabla_\\theta \\mathcal{H}$, see Eq. (20), and hence to evaluate third-order derivatives, which makes it more computationally intensive. \n\nTo address this challenges, GPI-PINN 2 uses an expectation-free version $\\mathcal{G}$ of the Hamiltonian $\\mathcal{H}$ and leverages the $L^2$-optimal property of conditional expectation (Proposition 2.3) to learn the value function from $\\mathcal{G}$, leading to the update rule (12). Combined with Proposition 2.2, this allows us to bypass the computation of the Hessian $\\nabla^2_x V_{\\theta}(t,x)$ and jump expectations $\\mathbb{E}^{\\mathcal{Z}}[V_\\theta(t,x+\\gamma(t,x,Z_1,a))]$, enabling faster numerical convergence for high-dimensional stochastic control problems with jumps. Moreover, GPI-PINN 2 only requires the backpropagation through the value network $\\nabla_\\theta V_\\theta$ and thus to evaluate at most second-order derivatives, see Eq. (23). \n\n\n**Q2:**  *The experiments for consumption-investment do not compare the proposed algorithm with other standard DL frameworks as done for the LQR setting. Could the authors share these experimental results as well?*\n\n**Answer:** Yes, sure. Regardless of the specific dynamics considered in the control problems, our GPI-PINN algorithms consistently achieve results that are 1-2 orders of magnitude more accurate than existing model-free RL methods. Similarly, the local approaches mentioned in the introduction, which rely on a time-discretization of the state dynamics, only yield results for a restricted set of pre-specified points and fail to provide a global solution across the entire time-space domain, *i.e.* for any $(t,x) \\in [0,T] \\times D$.  Finally, additional experiments comparing the performance of GPI-PINN 2 with existing baseline DL algorithms for the consumption-investment problem are ready and can be shared in the revised version of the manuscript.\n\n**Q3:**  *Numerical experiments seem to only address dimensions d=10 to 50 and are limited to LQR and consumption-investment problem. It would be nice to mention larger scale problems where this could be applied.  Could the authors comment on the scalability and potential of this approach to larger scale environments? Are there environments or settings that are particularly suited to the strengths of the proposed algorithms?*\n\n**Answer:** Thanks for the suggestion. Of course, it depends on the computational resources. But we can easily apply GPI-PINN 2 to control problems with up to 100-150 dimensions. On the other hand, in our comparison study, GPI-PINN 1 does not scale beyond 10 dimensions with jumps and 50 dimensions without jumps, which confirms the efficiency of GPI-PINN 2 as outlined in Figure 1 (see also the computational complexity analysis provided to Reviewer 1, Q5). Since we wanted to compare GPI-PINN 1 and 2, we only reported results for both GPI-PINNs up to 50 dimensions. However, higher-dimensional results for GPI-PINN 2 are available and could be included in the paper to further illustrate the scalability of the approach.\n\n\nAs long as the state dynamics of the control problem are known, GPI-PINN 2 offers an efficient and scalable approach that provides accurate solutions in high dimensions for both the value function and the optimal control at any point in the time-space domain $[0,T] \\times D$, while accommodating jumps. From a pure practical standpoint, it performs particularly well on LQR-type of control problems, regardless of the jump-size distribution. Note that even when the systems dynamics are not known, nothing prevents us from learning these dynamics in a preliminary step using some recent model-learning (RL) algorithm on observed data, with *e.g.* Brunton et al. (2016) or Champion et al. (2019). Once an approximate model is identified, GPI-PINN 2 can then be applied to efficiently solve the resulting control problem.  \n\n**Weakness 3:** We will improve the quality of the plots.\n\n - Brunton, S. L., Proctor, J. L., & Kutz, J. N. (2016). Discovering governing equations from data by sparse identification of nonlinear dynamical systems. _Proceedings of the national academy of sciences_, _113_(15), 3932-3937. \n- Champion, K., Lusch, B., Kutz, J. N., & Brunton, S. L. (2019). Data-driven discovery of coordinates and governing equations. _Proceedings of the National Academy of Sciences_, _116_(45), 22445-22451."}]

Author rebuttal:

Meta-review: This paper proposes a model-based actor-critic algorithm for solving a class of continuous-time stochastic control problem. The focus is on finite horizon problems. The method allows the dynamics to have jumps. The method benefits from the known dynamics to define PINN-style loss functions.

The paper introduces two methods: GPI-PINN 1 and GPI-PINN 2. The first method is not very efficient in dealing with jumps, while the second method is. The paper is theoretically motivated and shows the empirical effectiveness of the proposal methods.

The reviewers are mostly positive about the paper: we have one accept (5; confidence: 2), two borderline accepts (4; confidence: 3), and one borderline reject (3; confidence: 4).

On the positive side, the reviewers commented on the well-roundedness of the method in stochastic control theory, the generality of the method, empirical effectiveness, and that the paper is well-written, though sometimes dense in mathematical notation.
The major negative issue is that one step of the GPI-PINN 2 algorithm is not well-justified, as noted by Reviewer cXcY. There have been some discussions between the authors and the reviewer, but the issue is not fully resolved. The authors have a final message that they found a better theoretical justification for that part, but that was not explored further.

Without this justification, the paper basically offers two algorithms: one with all steps theoretically justified and another that is less theoretically justified in one of its steps. For both algorithms, we have empirical results showing their effectiveness. I confirmed with Reviewer cXcY that this issue was not critical. Overall, I think it is OK not to consider this issue a deal breaker.

I read the paper myself and I think this is a good paper. So, overall, I am recommending its acceptance.


I have one question: Equation (11) consider g to belong to the space of Borel measurable functions. This space is much larger than the space representable by NNs. This means that we may not obtain the minimum of the function. What is the effect of this on Proposition 2.3 and how the trick used based on it?
Decision: Accept (poster)