Title: Deep learning for continuous-time stochastic control with jumps
Venue: NeurIPS 2025
Accepted: yes
Keywords: Continuous-time stochastic control ; Jump diffusion ; Actor-critic ; Physics-informed neural networks ; Hamilton-Jacobi-Bellman equation ; Deep learning ; Bellman updating
Abstract: In this paper, we introduce a model-based deep-learning approach to solve finite-horizon continuous-time stochastic control problems with jumps. We iteratively train two neural networks: one to represent the optimal policy and the other to approximate the value function. Leveraging a continuous-time version of the dynamic programming principle, we derive two different training objectives based on the Hamilton--Jacobi--Bellman equation, ensuring that the networks capture the underlying stochastic dynamics. Empirical evaluations on different problems illustrate the accuracy and scalability of our approach, demonstrating its effectiveness in solving complex high-dimensional stochastic control tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a deep learning approach to solve stochastic optimal control problems with jumps. The method can be interpreted as a PINN-style approach to solving the HJB of the optimal control problem. Since the HJB is second order, a PINN style approach would require computing third-order derivatives, which quickly becomes infeasible. The paper proposes a second algorithm that circumvents this issue using a recursive approach instead of relying solely on HJB residuals. Theoretical results are accompanied with numerical results on small dimension settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Theoretical results are sound and proofs seem correct.\n2. The main algorithms provided are original and introduce a new approach to solving optimal control problems.\n3. The writing is thorough and comprehensive, and the mathematical results are precise.\n\nWeaknesses:\n1. Some of the exposition and notation is extremely dense and hard to follow.\n2. Numerical experiments seem to only address dimensions d=10 to 50 and are limited to LQR and consumption-investment problem. It would be nice to mention larger scale problems where this could be applied.\n3. The quality of the plots is unclear and makes it hard to read in some cases.\n\nQUESTIONS:\n1. Can the authors provide a condensed summary of differences between algorithm 1 and 2?\n2. The experiments for consumption-investment do not compare the proposed algorithm with other standard DL frameworks as done for the LQR setting. Could the authors share these experimental results as well?\n3. Could the authors comment on the scalability and potential of this approach to larger scale environments? Are there environments or settings that are particularly suited to the strengths of the proposed algorithms?\n\nLIMITATIONS:\nYes, the authors have adequately addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, past works in deep learning for continuous-time stochastic control are extended by utilizing physics-informed neural networks to reduce the computational burden of learning the optimal control and value function through generalized policy iteration. The first algorithm introduced, GPI-PINN-1, trains the value network to satisfy the Hamilton-Jacobi-Bellman equation by setting the loss function to be the residual. In this version the authors bypass computing the gradients and Hessian of the value function, but still need to evaluate the jump expectations in order to evaluate the Hamiltonian. GPI-PINN-2 not only bypasses the gradient computations of all orders, but also avoids explicitly evaluating the jump expectations by exploiting the known state dynamics, since the value target can be expressed in terms of these quantities. Furthermore, they use an iterative approach where the value network learned in the past iteration step is involved in computing the target for the new iteration step. However, this computational efficiency comes at the cost of reduced stability during training. The performance and training time of GPI-PINN-1 and 2 are compared, revealing that GPI-PINN-2 is by far the more practical algorithm when jumps must be accounted for despite its stability issues. Finally, numerical experiments on continuous-time stochastic control demonstrate the superiority of GPI-PINN-2 over existing model-free and model-based algorithms.\n\nSTRENGTHS AND WEAKNESSES:\n**Quality:** The techniques used in this paper have a solid theoretical foundation, and this is reflected in the strong empirical performance of GPI-PINN-1 and 2. The experiments run give a good idea of the performance benefits that GPI-PINN-2 can provide over GPI-PINN-1 and the baseline methods, validating that it would be worth adopting is many scenarios. Overall, I consider this to be high-quality work.\n\n**Clarity:** The overall clarity of the paper is good, with the algorithms being presented cleanly and the figures mostly being easy to interpret. However, there is a mismatch between the legend of the left plot of Figure 4 (which says the blue curve is the boundary loss) and the caption (which says the orange curve is the boundary loss). Furthermore, with the large amount of notation which needs to be introduced in the first two sections, adding a table in the appendix explaining each symbol may be a good idea.\n\nThere are a few questions which I had about the practical implementation and performance of the algorithms, which are given in the questions section.\n\n**Significance:** The problem formulation is very general and could be applied to many control problems, with the caveat that the system dynamics must be known. As the authors state, there are plenty of problems where certain dynamics can be modeled using known physical laws; however, in a real-life system the overall dynamics are likely to be influenced by less predictable phenomena as well. Furthermore, it is not obvious that a model trained in this way will be robust to noisy state measurements during deployment.\n\n**Originality:** This is the first application of physics-informed neural networks to this problem that I am aware of. However, the methods used are closely related to those introduced in [1], including the use of the least-squares Monte-Carlo method to bypass computing the jump expectation.\n\n[1] Duarte, V., Duarte, D., and Silva, D. H. Machine learning for continuous-time finance. The Review of Financial Studies, 37(11):3217–3271, 2024.\n\nQUESTIONS:\n**Q1:** How is the stopping condition in algorithm 1 evaluated efficiently?\n\n**Q2:** How high-dimensional can the problem get before GPI-PINN-2 struggles to solve it?\n\n**Q3:** What was the actual batch size per forward pass used when training the GPI-PINN networks? Was this batch size (and the number of batches per epoch) also used for the baseline deep learning algorithms?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates a finite-horizon continuous-time stochastic control problem with jumps. The associated Hamilton–Jacobi–Bellman (HJB) equation includes a nonlocal integral term due to the jumps, which makes the problem harder than normal optimal control problem. The authors propose a two-step iterative scheme: for a fixed feedback control, the value function is updated by minimizing the residual of the HJB equation; then, holding the value function fixed, the control is improved by minimizing the Hamiltonian. Two variants of the algorithm are introduced: the first uses multiple samples to approximate the jump expectation via Monte Carlo, while the second simplifies this step by using a single sample per spatial point. Numerical experiments are presented to demonstrate the performance and efficiency of both approaches. Overall, the paper is well-written.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n\nThe paper is well-written, with key concepts and algorithmic steps clearly explained.\n\nThe use of Proposition 2.2 enables an efficient approximation of the second-order differential operator without computing gradients and Hessians directly, which is beneficial for high-dimensional problems.\n\nThe second algorithm reduce computational costs.\n\nThe authors present detailed numerical experiments\n\nWeakness\n\nWhile the second algorithm reduces the computational cost, it lacks theoretical justification. I have put details in Questions.\n\nQUESTIONS:\nThe design of Algorithm 2 raises two conceptual concerns that should be clarified. I would suggest the paper be accepted if the authors could address this issue.\n\nIn the first part, you modified the PINN loss by fixing the parameter $\\theta$ in G and only optimize the term $V_\\theta$ in line 191.5. (I temporally ignore the boundary trem.) Reorganizing the terms, the bracket before taking square in loss $\\mathcal{L}_1$ can be viewed as\n\n$$V_\\theta – V_{\\theta^{(k)}} – RES_{HJB}(\\theta^{(k)}),$$\n\nwhere $RES_{HJB}$ is the residual of the HJB equation (before taking expectation) in the PINN loss. This update means we hope that the new value function is close to the old one added by the HJB residual. However, since $E[RES_{HJB}] = 0$ characterizes the true solution, there's no justification for using the residual itself as an additive update. In fact, using\n\n$$V_\\theta – V_{\\theta^{(k)}} + RES_{HJB}(\\theta^{(k)}),$$\n\n(the negative of the update) seems equally justified, leading to opposite update directions. The current construction appears arbitrary without additional justification.\n\nFor the second part, the Hamiltonian is expected to be maximized after taking expectation, while the algorithm only uses one sample for each x. Please explain why it is fine we only use one sample?\n\nBelow are some minor questions or comments.\n\n1.\tPage 2 line 71. Duan et al is not the original paper for TRPO. It is better to cite the original work.\n\n2.\tPage 3 line 104. HBJ should be HJB.\n\n3.\tPage 4 eq (11). One the left, should it be just a value function without the input?\n\n4.\tPage 5, part 2 of the first algorithm. I would suggest the authors add a remark that the term $\\partial_t v$ in $\\mathcal{H}$ does not involve I the optimization, so it is just minimizing the Hamiltonian. (At first glance, it seems you are updating the control using the residual of the HJB equation.)\n\n5.\tThe algorithm that consecutively update the value function and the control is very similar to the actor-critic methos in reinforcement learning. I think papers that implement this method in optimal control method such as\nSolving time-continuous stochastic optimal control problems: Algorithm design and convergence analysis of actor-critic flow\nCan be added to related work.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a model-based deep learning approach for solving finite-horizon continuous-time stochastic control problems that feature jumps. The proposed method iteratively trains two neural networks: one to represent the optimal policy and another to approximate the value function. Leveraging a continuous-time version of the dynamic programming principle, the authors derive two distinct training objectives based on the Hamilton-Jacobi-Bellman (HJB) equation, which aim to ensure the networks capture the underlying stochastic dynamics. They propose two algorithms, GPI-PINN 1 and GPI-PINN 2. GPI-PINN 1 minimizes HJB residuals, but is noted to be inefficient with jumps. GPI-PINN 2 is designed to circumvent costly computations of gradients, Hessians, and jump expectations, making it efficient for high-dimensional problems with jumps. Empirical evaluations on various problems, including linear-quadratic regulators and optimal consumption-investment problems (both with and without jumps), illustrate the accuracy and scalability of their approach, demonstrating its effectiveness in solving complex, high-dimensional stochastic control tasks.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper presents a rigorous and detailed model-based deep learning approach to a challenging class of stochastic control problems. Its strengths lie in its theoretical foundation and the proposed methods' efficiency for high-dimensional problems with jumps. However, there are some areas where the clarity and comprehensiveness of the presentation could be improved.\n\n**Strengths:**\n*  The paper tackles continuous-time stochastic control problems with jumps in high dimensions, which is a complex and relevant area. The development of GPI-PINN 2, specifically designed to handle jumps efficiently by avoiding computationally expensive gradient, Hessian, and expectation computations, is a notable contribution.\n* Unlike many model-free deep RL algorithms that rely solely on environmental sampling, this approach explicitly leverages known underlying dynamics (HJB equation). This design choice is claimed to lead to higher accuracy when state dynamics are fully known.\n* The iterative training of two neural networks for both the value function and optimal control provides a global approximation across the entire space-time domain, allowing for rapid online evaluation. This is an advantage over local approaches.\n*  The method's effectiveness, scalability, and accuracy are demonstrated across various numerical examples. These include LQR problems with and without controlled jump intensities, and complex optimal consumption-investment problems.\n* The approach is well-grounded in stochastic control theory, leveraging the HJB equation and providing propositions (2.2 and 2.3) that enable computational efficiencies.\n\n**Weaknesses:**\n* While propositions are provided, the transition from theoretical statements to the practical implementation within the algorithms (e.g., the exact form of $\\mathcal{G}$ in Algorithm 2's update rules) could be clearer in the main text. Some key definitions from standard literature are referenced but full clarity requires consulting external sources.\n* While the paper states that model-free RL algorithms are \"less accurate in cases where the state dynamics (2) are fully known and given\" and shows GPI-PINN 2 outperforming PPO and SAC in Figure 3, the explanation for why this is the case could be more deeply explored beyond \"rely solely on sampling\". A more nuanced discussion of the trade-offs (e.g., data efficiency, robustness to misspecification) would be beneficial.\n* The paper identifies that a limitation is the need to know the underlying dynamics of the state process, which is \"not always available in real-world applications\". While this is acknowledged, the implications for practical deployment are significant and warrant further discussion on how this model-based approach would integrate with real-world scenarios where dynamics need to be learned or estimated.\n* GPI-PINN 1's inefficiency with jumps is a major drawback. While GPI-PINN 2 addresses this, the practical utility of GPI-PINN 1 seems limited, and perhaps it should be framed more as a stepping stone to GPI-PINN 2 rather than a fully viable alternative.\n\nQUESTIONS:\n1.  In Algorithm 2, the function $\\mathcal{G}$ plays a crucial role in updating the value and control networks. Could the authors provide a more intuitive explanation for how $\\mathcal{G}$ relates to the original HJB equation and why its use helps circumvent the costly computations of expectations and higher-order derivatives? A deeper dive into the connection between minimizing $\\mathcal{G}$ and satisfying the optimality conditions would enhance clarity.\n2.  The paper highlights that model-free RL is less accurate when dynamics are known. Could the authors elaborate on how their approach (which relies on *known* dynamics) compares to recent advancements in model-learning within RL, where models are learned from data and then used for planning or control? This would provide a broader context for the \"model-based\" claim.\n3. The paper mentions various hyperparameters (e.g., $M_1, M_2, k_*, \\eta_1, \\eta_2$, $\\xi$, network architecture). A more detailed discussion on the sensitivity of the algorithms (especially GPI-PINN 2) to these parameters, and strategies for their robust selection across different problem instances, would be valuable for practical implementation. What are the criteria for selecting the proportionality factor $\\xi$ (L209) and its impact on convergence? \n4. The current formulation assumes specific characteristics for the random measure $N^\\alpha$ and the distribution $Z$ of jump vectors. How general is the approach for different types of jump processes (e.g., non-Poissonian jumps, jumps with state-dependent or control-dependent distributions that are more complex than the assumed form)?\n5.  While GPI-PINN 2 is stated to be more efficient, a more explicit computational complexity analysis (e.g., in terms of state dimension $d$, number of hidden layers $L$, and samples $M_1, M_2$) for both training and inference, compared to alternative methods, would solidify the claims about scalability.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-05"}