Title: A Temporal Difference Method for Stochastic Continuous Dynamics
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement Learning

Model-free RL

Continuous Control
Abstract: For continuous systems modeled by dynamical equations such as ODEs and SDEs, Bellman's principle of optimality takes the form of the Hamilton-Jacobi-Bellman (HJB) equation, which provides the theoretical target of reinforcement learning (RL). Although recent advances in RL successfully leverage this formulation, the existing methods typically assume the underlying dynamics are known a priori because they need explicit access to the drift and diffusion coefficients to update the value function following the HJB equation. We address this inherent limitation of HJB-based RL; we propose a model-free approach still targeting the HJB equation and the corresponding temporal difference method. We prove exponential stability of the induced continuous-time dynamics, and we empirically demonstrate the resulting advantages over transition–kernel–based formulations. The proposed formulation paves the way toward bridging stochastic control and model-free reinforcement learning.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a Reinforcement Learning algorithm that updates the value network using the stochastic HJB equation, thus bridging discrete-time and continuous-time insights. This method is tested on the MuJoCo environments and outperforms PPO.\n\nUnfortunately, although the experimental results are promising, I am concerned about the author's familiarity with HJB equations and important characteristics of the problem. I am willing to increase my score if the authors focus on making their work more grounded in the literature on viscosity solutions and continuous-time policy iteration in their rebuttal.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n* The paper is well-written and mathematically easy to follow. I followed the proof of Proposition 1 and the details on the HJB equation and believe both are correct. Appendix A.1 is correct and carefully grounded in the literature on stochastic differential equations. \n* Although Proposition 2 is based on problematic assumptions, I went through the proof and believe it is correct.\n* The derivation from the Bellman equation to the HJB to the dTD update is clear and well-motivated.\n* The experimental results seem promising. PPO with $\\beta$-TD significantly outperforms PPO in six out of twelve environments and shows comparable performance in the remaining environments (except half-cheetah)\n\n**Weaknesses**\n\n* I find Proposition 2 problematic. How are the assumptions verified? Assumption (i) is strong and may not hold in general. This undermines the generality of the contraction proof. $V$ will be, at best, locally Lipschitz in some sub-domains. Furthermore, it is not scalable: as n and m increase, $\\gamma$ needs to increase as well, which can severely limit the model's ability to consider long-horizon planning. How are $L_v$ and $L_\\sigma$ computed in numerical experiments?\n* In general, the value function will not have continuous first or second order derivatives - rather, it will be a viscosity solution (indeed, the study of viscosity solutions was motivated by the fact that the value function is not generally differentiable). For example, in LQR settings, the value function is not differentiable at the origin. I highly recommend that the authors read the paper [1] on neural network-based methods of mitigating these issues.\n* Fundamentally, the proposed method combines discrete-time and continuous-time solutions, which I believe is problematic. The proposed algorithm is based on PPO and thus uses the policy gradient (PG) theorem [2], which is derived in the discrete-time domain. If the authors want to use a policy gradient algorithm, they need to build their method based on the continuous-time variant of PG, for instance, pathwise policy gradient estimators [3].\n* Important baselines are missing: (1) Soft Actor-Critic [4], (2) HJ DQN [5].\n\n[1] Yiming Meng et al. \"Physics-Informed Neural Network Policy Iteration: Algorithms, Convergence, and Verification.\" ICML 2024.\n\n[2] Richard Sutton et al. \"Policy gradient methods for reinforcement learning with function approximation.\" NeurIPS 1999.\n\n[3] Remi Munos. \"Policy gradient in continuous time.\" JMLR (2006).\n\n[4] Tuomas Haarnoja. \"Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor.\" ICML 2018.\n\n[5] Jeongho Kim et al. \"Hamilton–Jacobi Deep Q-Learning for Deterministic Continuous-Time Systems with Lipschitz Continuous Controls.\" JMLR (2021)\n\nQUESTIONS:\n* In the abstract, \"Bridging stochastic optimal control and model-free reinforcement learning\", what is the difference?\n* In Appendix A.3, where you use Ito's lemma, you are missing $\\Delta t$ in some parts of the equation.\n* How is Proposition 2 used in the numerical experiments? How are $L_v$ and $L_\\sigma$ computed?\n\nLIMITATIONS:\n* I appreciate that the authors acknowledge some limitations (e.g., instability, off-policy inapplicability, large $\\gamma$) of their work.\n* As mentioned above, I suggest that the authors conduct an exploration of regions where $V$ isn't first or second-order differentiable.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an alternative algorithm to TD learning, called differential TD (dTD), for continuous-time, stochastic RL problems. Contrary to existing approaches from the optimal control literature, this algorithm is model-free (sample-based) like TD learning, but exploits the continuity of the value function. The authors demonstrate the practical potential of this approach, when mixed with classical TD iterations.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper, well-written and theoretically-grounded, tackles an interesting problem, in the relatively under-studied continuous setting in RL. The proposed algorithm is sample-based, hence compatible with RL problems. While it is limited to fixed-policy TD-based approaches, the paper shows with PPO how it can be applied to optimize the policy as well. The experiments are thoroughly led, with interesting discussions on the influence of the different parameters. The limitations are mainly related to the experimental comparison with the plain TD algorithm, where the actual advantage of dTD is difficult to assess in all situations. Here are more detailed limitations (see also questions below):\n- While the contraction analysis provided in Proposition 2 is important, a consistency result as $\\Delta t \\rightarrow 0$ is missing from the analysis. When dTD is applied with a finite $\\Delta t$, can we say anything about the error on the value function or the Bellman error?\n- In the experiments, the naive dTD and dTD algorithms have very different behaviors. However, since the  loss function is mentioned to be the MSE, I don’t understand how changing the target and prediction would change anything (up to a scaling factor of $\\gamma$)? This is the most confusing detail of the paper, and I think the precise difference between the two updates should be made much clearer.\n- TD is an algorithm designed for policy evaluation. Yet in the experiments, only PPO is tested, while TD is only a sub-element of it. Is it possible to assess the performance of dTD on policy evaluation only, where the distance to $V^{\\pi}$ can be directly evaluated? Maybe the influence of the different parameters would be easier to judge on this specific sub-problem.\n- As discussed in the paper, the optimal $\\beta$ depends on the problem. However, it is quite difficult to judge whether finding that some $\\beta >0$ performs better than $\\beta=0$ on some problems is indeed significant, or is only due to the fact that we are optimizing one more free parameter.\n\nQUESTIONS:\n- The assumptions of Proposition 2 are appropriately discussed in the paper. It is indeed known that $V$ has in general little regularity in the deterministic case. However I was wondering if something more could be said in the stochastic setting: the noise in the dynamics is usually seen as having a regularizing effect on the value function. See for example [Fleming and Rishel, *Deterministic and Stochastic Optimal Control*, 2012], where the Brownian motion makes the value function at least C^2 in the space variable.\n- In Eq. 8, a term $O(\\Delta t^{3/2})$ is missing from the rhs of the equation.\n- Is it possible to exhibit a toy example where exploiting the continuity of the value function is key, and makes dTD perform significantly better than plain TD? Did the experiments that you led give you any insight on the type of problems where dTD is expected to perform best? \n- I understand that mixing dTD with TD is better in terms of stability, however I believe that dTD (with $\\beta=1$) should be reported as well in the plots, even as a baseline. \n- Can you provide some more details on the parametric representation of $V$? Is it exactly the same for the TD, dTD and naive dTD algorithms? Is it adapted to the different problem instances?\n- Can you please clarify the exact difference between dTD and naive dTD, and why their performance is so different?\n\nLIMITATIONS:\nLimitations concerning stability issues and the large discount factor regime are appropriately mentioned in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn the present paper, the authors introduce differential temporal difference, a model-free approach to temporal differences in reinforcement learning. To derive the method the authors build on the Hamilton-Jacobi-Bellman (HJB) equation and use Ito's stochastic calculus to derive the algorithm for differential temporal differences from the HJB as well as a number of theoretical properties such as its contraction property. This approach is subsequently evaluated on 4 continuous control environments showing it outperforming the classical temporal differences approach.\n\nSTRENGTHS AND WEAKNESSES:\nThe main strengths of the paper lie in its super high clarity of the writeup throughout the paper, complementing its strong theoretical derivations. The step-by-step derivations starting out from the Hamilton-Jacobi-Bellman equation using the Ito's stochastic calculus are very clear, and easy to follow. The derived theoretical results on the contraction are a clear result of this strong theoretical foundation to the newly introduced algorithm.\n\nThe weaknesses of the paper all lie in its evaluation, to hone in on a number of the weaknesses:\n- The evaluation as is show a lack of comparison to other algorithms, while utilizing standard libraries like Brax, whose interface to reinforcement learning frameworks has seen wide use. This is especially surprising given the author's acknowledgement of model-based counterparts to the proposed model-free approach of the paper, which would readily lend themselves as points of reference on the performance of the newly introduced algorithm.\n- The evaluations as is, only compare to a limited number of control environments, where a larger number of (continuous) control environments are readily available through the same software packages used by the authors. \n- Claims with regards to the utility of the new algorithm, such as its potential use in PPO & A2C algorithm, are not reinforced through the evaluation design.\n\nQUESTIONS:\n### Main Questions (mostly re the evaluation)\n- As mentioned above, the evaluation currently lack comparison to other algorithms to better position the work of the present paper in relation to the wider literature. In the introduction you point out a number of previous works, orthogonal to yours (line 37ff). Have you considered adding e.g. model-based approaches to TD to the evaluations to improve the paper's comparability to wider literature?\n- _\"It is compatible with on-policy methods such as A2C (Mnih et al., 2016) and PPO (Schulman\net al., 2017)\"_ (line 44f), why is this not expanded upon with dedicated evaluations? As is, this potentially very strong evaluation claim is not backed up by the evaluation design.\n- With the theoretical derivations of the new algorithm being one of the major strengths of the paper, how derived theoretical properties utilized, or even leveraged in the evaluations?\n\n### Minor Questions\n- You position the paper as mostly useful for applications in robot learning & autonomous driving, do you think that your approach could potentially also hold a lot of utility for applications in the sciences?\n- Is the DEHB method used for hyperparameter tuning truly the most efficient hyperparameter tuning approach for reinforcement learning? The 2023 study referenced here as source of this knowledge is outdated. I would recommend to remove this half-sentence in light of the significant body of work on hyperparameter optimization since then. Furthermore, I would argue that their findings are no longer broadly applicable, at the same time not making the hyperparameter optimizer not all of a sudden a bad or wrong hyperparameter optimizer to choose.\n\nLIMITATIONS:\nLimitations only discussed implicitly throughout the paper, not explicitly in a dedicated _\"Limitations\"_ section. It would aid the reader considerably to summarize the limitations in one succinct limitations paragraph.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-09"}