A Temporal Difference Method for Stochastic Continuous Dynamics
Authors: Haruki Settai
Naoya Takeishi
Takehisa Yairi
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning
Model-free RL
Continuous Control
Primary area: reinforcement_learning
TL;DR: Reinforcement learning for systems described by differential equations: a model-free approach based on the HJB equation

Abstract: For continuous systems modeled by dynamical equations such as ODEs and SDEs, Bellman's principle of optimality takes the form of the Hamilton-Jacobi-Bellman (HJB) equation, which provides the theoretical target of reinforcement learning (RL). Although recent advances in RL successfully leverage this formulation, the existing methods typically assume the underlying dynamics are known a priori because they need explicit access to the drift and diffusion coefficients to update the value function following the HJB equation. We address this inherent limitation of HJB-based RL; we propose a model-free approach still targeting the HJB equation and the corresponding temporal difference method. We prove exponential stability of the induced continuous-time dynamics, and we empirically demonstrate the resulting advantages over transition–kernel–based formulations. The proposed formulation paves the way toward bridging stochastic control and model-free reinforcement learning.

A Temporal Difference Method for Stochastic Continuous Dynamics
Haruki Settai
Naoya Takeishi
Takehisa Yairi
The University of Tokyo
(sharuki, intakk staraatttaaaaaaaga....o ecc. wwt.c.u-tokkee u-tokyo. ac
Abstract
For continuous systems modeled by dynamical equations such as ODEs and SDEs, Bellman's principle of optimality takes the form the Hamilton-Jacobi- Dami-Bllmnn (HJB) equation, which provides the theoretical target of reinforcement learning (RL). Although recent advances in RL successfully leverage this formulation, the existing methods typically assume the underlying dynamics known priori because they need explicit access to the drift and diffusion coefficients to update the value function following the HJB equation. We address this inherent limitation of HJB-based RI; we propose a model-free approach still targeting the HJB equation and the corresponding temporal difference method. We prove exponential stability of the induced continuous- time dynamics, and we empirically demonstrate the resulting advantages over Kansition-Kemel-basee formulations. The proposed formulation paves the way toward bridging stochastic control and model-free reinforcement learning.
Introduction
Reinforcement learning (RL) ) has been successfully applied in various domains, ranging from discrete systems like board games (Silver et al. 2018) to systems that are continuous in both state and time such as robotic control (Kober al. 2013). RL research has been advancing through a variety of approaches, and much of the work has focused on improving methods by refining objective functions (Schulman et al. 2015 2017), balancing exploration and exploitation (Haarnoja et al. 2018), and developing more effective architectures (Hafner al. 2019). These efforts have significantly advanced the field, leading to the development of various successful algorithms.
In contrast to the studies primarily targeting algorithmic components or optimization techniques, we focus on the continuity time and explore what we call continuous RL, where the dynamics of systems are described by ordinary differential equations (ODEs) stochastic differential equations (SDEs), which relatively underexplored aspect of RI.. Although many RI. methods have been applied, sometimes heuristically, to both discrete and continuous systems, the continuity of the system has not necessarily been fully exploited, even when known advance. Laying foundation for utilizing prior knowledge of continuity is important toward more effective learning and decision- making methods, particularly for practical applications such as robot control and autonomous driving, which typically fall into the target of continuous RL.
way to incorporate prior knowledge of the system's continuity into the learning process is to use the Hamilron-JacobiBBe Bellman (HJB) equation. In the Bellman equation, continuity is encoded in the transition kernel. However, in model-free RI,, where transitions are approximated using samples, this continuity information is lost because the transition kernel is not explicitly modeled. On the other hand, the HJB equation retains the continuity information in the argument of the expectation rather than in the transition kernel. This property allows the HJB equation to keep taking continuity into
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

account even under sample-based approximations. However, because the HJB equation depends on the coefficient functions of the system's dynamics, prior work has largely been limited model-basee approaches (Munos and Bourgine
1997 Yildız et al. 202T).
In this paper, we introduce temporal difference (TD) method hased the HJB equation, namely differential TD (dTD), achieved through sample-based approximation of the expectation term in the HJB equation. dTT enables policy evaluation without requiring knowledge or estimation of the system dynamics, while incorporating the continuity of the dynamics into the learning process. is compatible with on-policy methods such as A2C (Mnih et al. (016) and PPO (Schulman et al. 2017), and we demonstrate its effectiveness on Mujoco (Todorov et al. (0122) tasks including Hopper, HalfCheetah. Ant, and Humamoid. The codes for the proposed method are available at https://ght thub. cow/4thhia/ com//tahiaadlffeerriiaaaa. differentialllIDD
2 Related Work
Deterministic Dynamics The study of continuous RL for ODE systems can be traced back to studies such as BBird (1994); [Mnnos [197); Doya (2000; Munos (2006). [aird (9944) discovered that the Q-function collapses in continuous RL, which was rigorously proven and extended to deep RL in Tallec et al. (201)). In Munos (997), model-free approaches for ODE systems were first studied. Doya 2000) was the first to introduce TD for ODE systems and extended it to TD(A) and actor-eritic. Munos (2006) investigated the estimation of policy gradients for ODE systems and proposed pathwise derivative approach. More recently. VVmvoudakis and Crofton (017)) developed a model-free RL framework for deterministic linear systems. Kim et al. (2021) proposed a model- Q-learning approach in which the control is derived from the HJB equation, while the learning target is based on the conventional Bellman equation. Yıldız et al. (202I) introduced a model-basee method that leverages the Neural ODE framework to enable continuous-time optimization using learned system dynamics.
Stochastic Dynamics One of the earliest works on RL in SDE systems is Munos and Bourgine (1997). which takes a model-based approach. However, research in this direction remained largely unexplored until recently. In the past few years, a growing body of work has emerged that aims to establish theoretical foundations for RL in stochastic dynamics. [Wang et al. (2020) introduced an entropy-regularized relaxed control formulation and provided a comprehensive analysis in the LQR setting. (Tng et al. (2022) further demonstrated the well-posedness of the HJB equation within this relaxed control framework. Jia and Zhou (2022b) showed that Bellman optimality is equivalent to maintaining the martingale property of a suitably defined stochastic process. and proposed a corresponding algorithm. Building on this approach. Jia and Zhou (2022al 2023) proposed actor- critic and Q-learning algorithms for finite-horizon SDE systems, respectively, Zhao et al. (2020) extended key theoretical tools such as the state visitation distribution and the performance difference lemma to the continuous-time setting and applied them to TRPO and PPO. Despite these advances, most of the above approaches (i) operate in a model-based regime and, in addition, (ii) are restricted to mnite-horizon settings that require access to full trajectory information and/or (iii) assume linear dynamics model. In contrast, our work proposes a simple model-free TD method that is applicable to general nonlinear stochastic dynamics without requiring knowledge of the SDE coefficients. We demonstrate its effectiveness on standard certimoousconntre RL benchmarks, going beyond prior work that has been mostly confined to toy SDE examples.
3 Background
3.1 Problem Setting
We consider a continuous-time RL setting where the state space is S C R" and the action space is A.
We model the state dynamics by the following controlled SDE:
dS.- u(S;, At)dt (SS
(I)
where u S x A - R S S A - R and (B:) is the m-dimensional Brownian motion. Note that the state evolution is influenced by both the inherent noise in the system as well as the randomness induced by the stochastic policy : S + P(A), where P(A) is the space of probability
2

---

distribution over the action space. Thus, the expectation related to this SDE is expressed as Ep, [.] where Pr denotes the transition kernel induced by ( For simplicity, we assume that the stochastic process ① is well-defined; see Appendix A. for detailed justification. We here focus on SDF systems because we can recover results for ODE systems in the limit of a =
We primarily focus on the model-free setting, where the agent has no prior knowledge of the dynamics coefficients μ. and in Eq. ( Throughout this paper, the term RL refers this model-free setting unless specifically stated otherwise (c.g., as model-based RL).
3.2 Continuous RL
The dynamics governed by the SDE in Eq. ① exhibit the Markov property (informally, the infinitesi- mal evolution depends the past only through At)), and thus continuous RL falls within the general framework of an MDP. For example, the Bellman optimality equation can be written as follows:
(*():) = max Ep. f - p(S, A,) te -0v' S: = St
where p: S x A -> R is the reward rate function, and (0, oc) is the constant discount rate. Here, (*(s) the optimal value function, defined as:
(* == max Ep, V.= ( Ar)dr S;
When learning based on discrete observations of continuous system, such as simulations, one can discretize the problem follows:
V (s:) max Ep. Az) + - "*AVV
(2)
where At is a small time interval and need not be constant. This shows that standard RL methods can be applied continuous RL, although the use of Q-functions in continuous time requires care (Tallec) et al. (019). However, this discretization-bu based approach can be inefficient in the model-free setting, where the expectation approximated using samples. In particular, the system dynamics enter only through the transition kernel, and approximating the expectation by samples does not explicitly exploit the underlying continuity of the dynamics. As a result, the agent may fail leverage structural information that is specific to continuous-time systems.
3.3 Natural Target of Continuous RL
We focus on TD learning. a fundamental approach in RL. TD learning naturally targets two types of Bellman equations: the Bellman optimality equation and the Bellman expectation equation. Combining these with the choice of value function (V or Q) yields four candidate targets. Among them, the V-Bellman optimality equation is immediately excluded as it is not compatible with model- free framework. Moreover, as we will show in Section 4 the Q- Q-Bellman optimality equation is challenging to learn effectively with TD methods in continuous RL due to the maximization operator. Thus, in continuous RL, policy evaluation is the most natural scope.
This leaves two remaining candidates: the V- and Q-Bellman expectation equations. While the Q-Bellman expectation equation is compatible with our methodology, Q-functions often degenerate in continuous RI. (Tallec et al. 2019), introducing unnecessary complexity. We therefore focus policy evaluation via the V-Bellman expectation equation,
v (8t) Ep,, [p(se, +e ("
3.4 Contrast with Stochastic Control
Much of the prior work on continuou RI. originates from the field of stochastic control. These approaches often start by directly introducing the HJB equation, which the continuous counterpart of the Bellman optimality equation (2). For instance, the HJB equation can be expressed as: (sec

---

Appendix A.2 for more detail):
(8)) = max En p(5,, Ad) + Σ n² (sts A1)
1
(s)
7
Ost
(s)
(a Asjo (st, A:)
as
where jd and (0o T denote the i-th element of y and the (i, (-th element of oo respectively.
However, introducing the HJB equation continuous RL typically restricts the methodology to model-based approaches, since the update requires explicit access to and (and the same issue persists even when using Q-functions; see Section 4 Thus, for TD learning, the HJB equation offers no clear advantage over the standard Bellman formulation (
In contrast, our goal is design model-free algorithms that explicitly leverage the continuity of the underlying dynamics, which naturally leads us to introduce a variant of the HJB equation built upon the V-Bellman expectation equation
TD Method for Stochastic Continuous Dynamies
A natural way to inform the agent that the system follows an SDE is to embed the model directly into the update rule. As discussed Section with the standard Bellman expectation equation (3) this information is lost when we pass to sample-based approximation, which drops the expectation subscript pr where the model is encoded. Therefore, by encoding the SDE information in the
argument rather than in the subscript of the expectation, one can expect to keep this information from being lost even under sample approximation. This can be achieved by further transforming the Bellman expectation equation using the SDE, i.e., by expanding (()+ via Itô's 's formula, resulting a variant of the HJB equation (the derivation is identical to that for the HJB equation; see Appendix A.2 for more details):
(*.).)
1 Er p(st, A.)
Σ (se, Ar)
(v)
(st, A.) lo A
(V)
Oz
Os'Ost
==1==1
We call this equation the HJB for fixed policy.
Now, are we all ready to implement model-free free value iteration just by approximating the expectation on the right-hand side of (5) with samples? The answer is because the argument of the expectation includes the coefficient functions of the SDE, μ and a,, making it impossible to directly approximate the expectation with samples.
4.1 Deriving TD from the HJB equation
We now present our main theoretical result. gives the foundation for our model-free formulation of sumporal-difference learning based on the HJB equation. The idea is that the drift and diffusion terms the HJB equation can be equivalently expressed using limits of sample-based finite differences. Proposition 1. When a stochastic process (S follows the SDE in 1 we have
SKi
Ep [1'( Ar)]
lim 1p At-0
(6)
At
and
(SL
(SttAr
()) st)
Ep [[o A:jo T A+)]"]] lim E
At-5
At
Proof. For the first claim, we expand the i-th component of the SDE I using the Ito formula:
Su =st + p' At) +
(o A.) B() + O(At2).
(8)
j=1

---

V(se
V(S++ ar)
Figure 1: Qualitative difference between the typical TD method and the proposed dTD method; the objects in red indicate what is adjusted by each temporal differ- ence. (Left) In the typical TD method, the values of V are adjusted TD error. (Right) In the dTD method, the gradient and the second derivative of V at 9t are adjusted minimize the dTD error.
minimize the
V(S.)
v(sg) proposed dTD
typical TD
Since B B+ Bj follows zero-mean Gaussian and is independent of the state and the action,
m Σ
m Σ ==1
Eyp
(s,, A:) - BB) 1=1
[a'
Ao) E B B Bg] =
By taking the expectation of both sides of (8) and then letting At + O, the terms in (( vanish, and we obtain (6). For the second part of the claim, we begin by considering the product:
(Si+ar +))(S( s()
= 1'(s,, A (s + gik (s1, (
(B)+ B.)
+ + AD
Σ o A.) + Ad) Σ (t A:)(B -B)) + O(A33).
(9)
and then take the expectation of both sides. Using the fact
E [[ (B BB)] = St
where dig 1 if l and det otherwise, we can take the expectation of both sides of equation (9) and then let At O, which yields 7
D
Remark Note that since Stt in equations (5) and (6) is sampled under the policy n, our method
applicable off-policy setfings such value iteration Q-learning. This limitation reflects fundamental distinction: our formulation relies on the HJB equation under fixed policy (i.e., policy
evaluation), rather than the classical HJB equation involving maximization over all policies.
From Proposition I the HJB equation 5 can be reformulated as
8 (s) At
(*t(s)
lim E 7 A140
p(s,, A.)
At
Os
18c
(S
x)) - st) (x)
2
At
as
As we have rearranged the HJB equation so that the argument expectation does not depend on construct lemporal-differeme update directly from it. We refer to this
the model, μ and update as differential temporal difference (dTD) and expect to be particularly effective when the observation interval At is small.
Definition 1 (differential temporal difference). Let At > O be time step and V denote an estimated value function. The dTD is defined as:
84 (V(s)
ATD
p(st, a.) )+ + At Os
(10)
n
(8/+2)
(8)
s1) x4) 82
V(s)).
At
Os (.)
5

---

As illustrated in Figure unlike conventional TD methods based on transition kernels, dTD en- courages the learning smooth value function by incorporating the continuity of the state space, even under sample-based approximation. We note that the version of dTD for ODE systems can be recovered by simply removing the term corresponding the diffusion coefficient a.
4.2 Convergence Analysis
A standard TD convergence analysis relies on the Bellman operator being a contraction. In contrast, the fixed-policy HJB operator involves unbounded differential operators and is a contraction in general, making the usual arguments inapplicable. Moreover, parameter-space analysis is technically challenging since the update depends on state derivatives of the value function. Therefore, we adopt an idcalized function-space analysis and study the continuous- time limit of the induced dynamics using PDE techniques. For the convergence analysis, we rewrite the second-order term in matrix form for notational convenience.
We first define the key operators for the fixed-policy HJB equation. Definition (Fixed-policy HJB operator). Given stationary Markov policy discount rate > 0, and functions Ja, o. p. define the policy-averaged coefficients
A)o D(s) = (s (s, The fixed-policy HJB operator ' maps V S - - to
p(s) == E (p(),
p(s) = 1s A)],
1 (pl)) (TV) ?
where L the infinitesimal generator defined below:
Definition 3 (Infinitesimal generator). The infinitesimal generator L maps any C2 function V to
(=) VV(s) 1 ()((())VVV()))
We analyze the iterative scheme
Vi++ V/ 7h (TVx Vk))
through its continuous-time limit
@v(() -V Ø1
which corresponds idealized setting where the function V be updated directly function space. Convergence is characterized by asymptotic stability of the unique fixed point satisfying TV V. Since we consider fixed policy (and thus no maximization operator), this fixed-point equation reduces to linear elliptic PDE. Defining the expected reward rate p(s) := (E A)), the equation V TV is equivalent to
(TI LV) p(s).
We then apply standard elliptic PDE theory (via the Lax -Milgra theorem) to the bilinear form associated with (yl - L which guarantees existence, uniqueness, and stability of the solution, and hence convergence of the induced dynamics. We leave a full analysis under function approximation to future work.
Assumption 1. We assume:
1. (Domain) S C R. is bounded with 2 boundary ØS.
2. (Coefficients) p L2).). The diffusion D (S; is symmetric, and the effective drift b = ā div D belongs to L (S;
3. (Uniform ellipticity) There exists such that T D($) a a|ll| for all E S, R..
4. (Reflecting boundary) The Neumann condition DVV 0 holds OS.
5. (Coercivity discount) 7 1b1l3//a.

---

We then apply standard elliptic PDE theory (via the Lax Mallgam theorem) to the bilinear form associated with (>/ _") ) guarantee existence and uniqueness of the fixed point. The exponential stability of the induced continuous-time dynamics is established separately below.
Lemma 1 (Existence and Uniqueness of the Fixed Point). Under Assumption the linear elliptic PDE
(+I (+I---)V(s)= Ā(s) for E
admits unique weak solution V E (S)-
Proposition (Exponential Stability of the Dynamics). Lei V (S)S) be the unique weak solution of (T L = Under Assumption !
the solution V(1) of
@v TV(t) - V(t)
δt
satisfies, for any V(0) E (S),
[V((t) - V V''11 (3()) I eepp Xt) |V V (((())
where one may take
A (
|
> 0,
b: ş div D.
a
5
Method
This section outlines our method for applying dTD in deep reinforcement learning. Because dTD relies on function approximation, we restrict our attention to the deep RL regime, representing the value function with neural network. A concise pseudocode listing is provided in Appendix B.I here we explain the loss formulation and the 3-dTD stabilization strategy.
5.1 Loss function
In deep RL,, TD methods typically use fixed target, known as the TD target, r(ss, t) + "diseress ((+++1) as the teacher and aim approximate the prediction (s²) by minimizing the squared error between them. Although it may seem natural, by analogy with classical TD, to treat the V(st) term in the dTD (T) the prediction and regard the remaining terms as the dTD target, this is in fact unnecessary. As shown in Appendix A the terms that appear in (5) and thus in (IO) are derived through series of transformations, and thus in dTD we no longer interpret the terms other than (st) as a low-variance estimate of the Bellman error. Consequently, the split between prediction and target is design choice.
We examine two different ways of defining the prediction and target from the rhs of 10P
As baseline, we first consider naive formulation following the typical TD-style decomposition, that is, we treat ((s) as the prediction. We refer to this approach as naive-dTD..
On the other hand, since TD methods are intended learn the value function V rather than its derivatives, we treat the value-based terms as the target and regard the derivative-based terms as the prediction. We term such parametrization simply as dTD hereafter.
These two variants are summarized in Table [ As introduced in the next section, we empirically found that dTD performed significantly better than naive-dTD.
5.2 Hybrid scheme for stabilizing dTD
Although Proposition] 2 establishes convergence analysis, it relies on an idcalized setting and does not cover the practical instability that can arise from function approximation errors in deep RL. Consequently, we cannot priori guarantee that plain dTD operates stably and efficiently in practice.
To make the critic update more robust, we linearly combine the classical TD error with the dTD error, using weights 1 - 3 and B, respectively; we call the resulting update 3-dTD. The TD part supplies the empirical stability that underpins most deep RL algorithms, whereas the dTD part injects gradient information from the continuous dynamics, accelerating learning when the underlying assumptions are approximately satisfied. We hypothesize that 3-dTD can strike a balance to stabilize learning progress and potentially improve convergence behavior in practice.
7

---

Table Comparison of target and prediction terms in TD methods, Here, Asj 8 ot denotes the i-th component of the state transition over a small time interval At.
Target
Prediction
TD
188 1) + 7dsscrtte (88-1)
V(*)
(s)
p(s,, ax) +
A
Ost
naive-dTD
n
n
AstAs (
(V(+))
2
At
0s
Ast (s
At
Os
dTD
- -(s), + (V(+444)
n
Ast A (V
At
0s
Experiments
6.1 Modification for discrete environment compatibility
In our theoretical framework, we work with continuous rewards (i.e., reward rate function) and specific form of the discount ratio e_>, which is not directly compatible with the discrete discount ratio "diserete- To address this, we adjusted the reward and discount ratio following the same approach discussed in Tallec et al. (2019). The continuous reward formulation can be approximated by:
72 p(s az dt
R
In this approximation, p(s,, At corresponds to the discrete reward and e e- corresponds to the discrete discount ratio "Jdiscrete- Thus, we can establish the following relationship:
r() and
p(s a.)
7 = -=-A At
log log((diserww))
At
This adjustment ensures that the observed discrete rewards are properly scaled to align with the continuous reward formulation used in the dTD method. With this scaling, dTD can be computed as
dTD Target: -r) log(Tdiserene) V (S(+4t))
and
7 E stt (v(s)
n
VV)(s) 0s;0s;
dTD Prediction
ΣΣ
( At
st)(s
st)
0%
6.2 Experiment design
Environment We conducted experiments with the Brax library (Freeman et al. 2021 in the following environments: Hopper, HalfCheetah, Ant and Humanoid. Each environment provides mid- to high-dimensional state space, with the number of state components varying across environments: Hopper (11 dimensions), HalfCheetah (17 dimensions), Ant (27 dimensions) and Humanoid (244 dimensions). In each environment, every step, perturbed each state component by adding noise in the form of
8; - + coef |s:| noise,
where noise (0, 1), and tested for three values coef 0.00, 0.01, 0.05. By adding this process noise, we aim to simulate with SDE systems, with the case of coef = 0.00 representing the limit case corresponding to ODEs. The specific time-step values used for each environment, which are directly used in the learning process but are important ensure they are small enough, are: Hopper: A/ 0.008, HalfCheetah: Al 0.05, Ant: At 0.05 and Humanoid: A/ = 0.015.
http:////github..ommogggg//////

---

-10
Hopper (noise: 0.00)
-103
Hopper (noise: 0.01)
-103
Hopper (noise: 0.05)
4
Episodic retur
2-
2
0
0
10
2
10
D
2
110
100
Halfcheetah (noise: 0.00)
Halfcheetah (noise: 0.01)
-10
Halfcheetah (noise: 0.05)
2
Episodic return
1
0
100
-106
2
-00
-10
Ant (noise: 0.00)
-10
Ant (noise: 0.01)
-10
Ant (noise: 0.05)
Episodic return
4
2
2
0
0
-106
-106
2
-106
-102
Humanoid (noise: 0.00)
-10
Humanoid (noise: 0.01)
-10-
Humanoid (noise: 0.05)
Episodic return
4
2-
2
.106
-106
100
Total episode step
Total episode step
Total episode step
Figure 2: Performance of TD, 3-naive-dTD, and 3-dTD on continuous control benchmark. These results were obtained using the PPO algorithm. Each column corresponds to different noise levels (coef = 0.00, 0.01, 0.05), and each row corresponds to different environments. The tuned 3 values for 3-naive-dTD were 0.08, 0.07, 0.23, 0.02 and for 3-dTD were 0.57, 0.74, 0.24, 0.33 in Hopper, HalfCheetah, Ant, and Humanoid, respectively.
Baseline Various methods have been developed specifically for settings such ODE Tallec et al. 2019), LQR (Vamvoudakis and Crofton (017), or time-dependent Q functions with finite horizons (Jia and Zhou [202)), but often incompatible with the current deep RL framework (Kim et al. 2021) or rely on prodel-based assumptions (Munos and Bourgine 1997). making them unsuitable for comparison with our proposed method. We chose
standard TD methods baselines and experimented with TD, 3-naive-dTD, and B-dTDD using A2C(Mnih et al. 2016) and PPO (Schulman)
al. 2017). In this paper, present the results using PPO as the primary focus, and the results using A2C are provided in the Appendix B.4 for supplementary details.
Hyperparameter tuning For hyperparameter tuning, applied the DEHB (Awad al. [02]] l). a multi-fidelity method that currently considered the most effective method RL (Eimer et al. 2023). While performed hyperparameter tuning for the standard PPO algorithm well, also reference the official tuning results from Freeman et al. (2021) for fair comparison. Additional details about the hyperparameter search space can be found in Appendix B
6.3 Results and discussion
Comparative evaluation We compare (the variants of) the proposed method and the bascline:

---

(6-naive-dTDD vs. B-dTD) In Figure we can observe that 3-dTD consistently outperforms S- naive-dTD. all the environments, the optimized values of for 3-naive-dTD were quite small, suggesting that the effective update rule of (-naive-dTDD became close to that of the standard TD. Despite such a fact, however, the performance of 3-naive-dTDD remains significantly worse than the standard TD. There are two possible explanations: (1) the ,3 value chosen for (8-naiveddTD was actually still not small enough to fully eliminate the adverse effect of the naive-dTD term; and (2) the TD-related parameters in 3-naive-dTD were only suboptimally tuned because hyperparameter tuning resources were allocated mainly to optimizing 3. These factors may jointly account for the unexpectedly poor performance of 3 D-naive-dTD.
(TD vs. (-dTTD) As shown in Figure 3, G-dTD outperforms TD or achieves comparable performance in all cases. While the degree of improvement varies, the final performance of TD and 3-dTD tends to converge, which is not very surprising because both dTD and TD are derived from the same Bellman equation, and the resulting value functions should thus be similar to each other eventually. Nevertheless, dTD has the advantage of implicitly utilizing continuity information during training, which enables it to make more informative updates. Consequently, although the final performance may be comparable, (-dTD tends to show faster rate of improvement relative TD.
Significance of dTD In contrast to 3-naive-dTD, the weight / in 3-dTD is not exceedingly small. Notably, in the Halfcheetah environment, 8 assumes a relatively large value of 0.74. This indicates that dTD retains a meaningful impact on the learning process.
Impact of process noise In terms of robustness to process noise, both 3-dTD and TD exhibit similar performance. When the noise level is coef = 0.01, neither method experiences significant degradation performance. However, when the noise level is increased to coef 0.05, both B-dTD and TD show similar reduction in performance, particularly in environments like Ant and Halfcheetah.
7 Conclusion
We have presented differential TD (dTD), a temporal difference method based on the HJB equation. In contrast to approaches based transition kernels, the proposed method can incorporate the continuity of dynamics into the learning process without knowing the dynamics. We have shown empirical results for a variety of continuous control environments with different time intervals. The empirical results highlight the potential advantages of dTD in terms of learning speed and efficiency while also implying that stability concerns may exist in practice, which led to the introduction of the robust 3-dTD update. Although the current paper focuses on the theoretical development of dTD, these observations are useful and also warrant further empirical exploration.
We have also analyzed the conditions under which the continuous time dynamics of the HJB equation exhibit exponential stability toward the unique fixed point. This stability property, proven using techniques from linear elliptic PDE theory, is crucial for showing the theoretical convergence of the idealized iterative scheme. However, drawback is that the sufficient conditions we identify, such as the requirement for a bounded domain and uniform ellipticity (Assumption I]. are often hard to maintain or verify in the context of deep RL with function approximation.
Future work includes bridging the gap between the theoretical exponential stability and the practical stability of dTD updates (e.g., by ensuring the coercivity condition in practice), reducing the variance of learning by improved estimators or regularization, and extending the wide range of existing TD-based techniques to the dTD framework.
Acknowledgements
NT was supported by JST PRESTO JPMJPR24T6, JP20K19869, and JSPS JP25H01454.
References
Noor Awad, Neeratyoy Mallik, and Frank Hutter. DEHB: Evolutionary hyperband for scalable, robust and efficient hyperparameter optimization. In Proceedings of the 30th International Joint Conference Artificial Intelligence, pages 2147-2153, 2021.
10

---

Leemon C. Baird. Reinforeement learning in continuous time: Advantage updating. In Proceedings of 1994 IEEE International Conference on Neural Networks, volume 4, pages 2448- 1448-2453, 1994.
Kenji Doya. Reinforcement learning in continuous time and space. Neural Computation, 12(1): 219-245, 2000.
Theresa Eimer, Marius Lindauer, and Roberta Raileanu. Hyperparameters in reinforcement learning and how tune them. In Proceedings of the 40th International Conference on Machine Learning, pages 9104 9104-9149, 2023.
Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax a differentiable physics engine for large scale rigid body simulation. In Proceedings of the Neural Information Processing Systems Truck Datasets and Benchmarks, volume 1, 2021.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pages 1861-1870, 2018.
Danijar Hafner, Timothy Lillicrap, and Jimmy Ba. Dream to control: Learning behaviors by latent imagination. arXiv: 1912.01603, 2019.
Yanwei Jia and Xun Yu Zhou. Policy gradient and actor-critic learning in continuous time and space: Theory and algorithms. Journal of Machine Learning Research, 23(275), 2022a.
Yanwei Jia and Xun Yu Zhou. Policy evaluation and cmmporll-differee learning in continuous time and space: A martingale approach. Journal of Machine Learning Research, 23(154), 2022b.
Yanwei Jia and Xun Yu Zhou. q-learning in continuous time. Journal of Machine Learning Research, 24(161), 2023.
Jeongho Kim, Jaeuk Shin, and Insoon Yang. Hamilton-Jacobi deep Q-learning for deterministic continuous-time systems with Lipschitz continuous controls. Journal of Machine Learning Research, 22(206), 2021.
Jens Kober, J Andrew Bagnell, and Jan Peters. Some studies in machine learning using the game of checkers. The International Journal of Robotics Research, 12(11):22388) 2013.
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Tim Harley, Timothy P. Lillicrap, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proceedings of the 33rd International Conference on Machine Learning, pages 1937, 2016.
Rémi Munos. A convergent reinforcement learning algorithm in the continuous case based finite difference method. In Proceedings of the 15th International Joint Conference on Artificial Intelligence, volume 2, pages 826-831, 1997.
Rémi Munos. Policy gradient in continuous time. Journal of Machine Learning Research, 7(27):
771-791, 2006.
Rémi Munos and Paul Bourgine. Reinforcement learning for continuous stochastic control problems. In Advances in Neural Information Processing Systems, volume 10, 1997.
John Schulman, Sergey Levine, Philipp Moritz, Michael Jordan, and Pieter Abbeel. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning, volume 37, pages 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv: 7777006377 2017.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Si- monyan, and Demis Hassabis. Mastering chess and shogi by self-play with general reinforcement learning algorithm. 1.1::1712.01815 2018.

---

Corentin Tallec, Léonard Blier, and Yann Ollivier. Making deep Q-learning methods robust to time discretization. In Proceedings of the 36th International Conference on Machine Learning. volume 97, pages 6096-6104, 2019.
Wenpin Tang, Paul Yuming Zhang, and Xun Yu Zhou. Exploratory hjb equations and their conver- gence. SIAM Journal on Control and Optimization, 60, 2022.
Emanuel Todorov, Tom Erez, and Yuval Tassa. MuloCo: A physics engine for model-based control. In 2012 JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 5026-5033. IEEE, 2012.
Kyriakos G. Vamvoudakis and Kevin T. Crofton. Q-learning for continuous-time linear systems: A model-free infinite horizon optimal control approach. Systems & Control Letters, 100: 2017.
Haoran Wang, Thaleia Zariphopoulou, and Xun Yu Zhou. Reinforcement learning in continuous time and space: A stochastic control approach. Journal of Machine Learning Research, (198), 2020.
Çagatay Yıldız, Markus Heinonen, and Harri Lähdesmäki. Continuous-time time model-based reinforce- ment learning. In Proceedings of the 38th International Conference on Machine Learning, pages 12009-12018, 2021.
Hanyang Zhao, Wenpin Tang, and David D. Yao. Policy optimization for continuous reinforcement learning. Advances in Neural Information Processing Systems 36, pages 13637-13663, 2020.
12

---

A Mathematical Details
A.1 Justification for the Continuous RL Formulation
In Section 3, we modeled the evolution of the state under a stochastic policy by the controlled SDE
ds. ((S,, A.) di aS(,, Ar) A ~T((S)))))....
Here, the control applied in the form of action samples drawn from stochastic policy at each time step. While this formulation closely reflects the sampling- based behavior in RL, it raises technical challenge: the presence of external randomness in addition to the intrinsic Brownian noise introduces analytical difficulties. As result, the well-definedness of this SDE is not immediately obvious.
To address this issue, many prior works (e.g., Wang al. (020 Tia and Zhou (2022b a 2023); Zhao
al. (2020) adopt the averaged dynamics, denoted by (S0)22 whose distribution at each time
known to coincide with that of the original one under the same initial condition (Wang et al. (0200)
Specifically, the averaged dynamics defined as
ds. TTSS, x) + +),,,,(
where m)(, -Aar) ())s,, a(s = (S4 (s) (s))))( and (B,) is the m-dimensional Brownian motion. Since the averaged dynamics no longer involves the external randomness induced by stochastic action selection, its well-definedness is ensured by classical SDE theory under standard assumptions such as Lipschitz continuity and a linear growth condition.
Since the marginal distributions of the two dynamics coincide, the corresponding value functions also coincide:
(T(() = Ep,
c -- p(S Ar) dr
S, =
1 -- () 7(3 (, 7) dr 5 =
Egg
vs(ss)
where ((. SA pls, ((a) da. Hence the value function above is itself well defined and raises no analytical issues.
A.2 Ito formula
The Bellman equation is given by:
V*(a)) max Ep, [p(st, At)A te -
13

---

Assuming that stochastic process (Si)120 follows the SDE (1), the term V (S(+ar) can be further expanded using Itô's lemma:
(*)) max Ep, AI) +
max Eps p(s,, A, +0 THI V+((s)) +
Σ
1¹ (00, Ar) Ab)
(s)
Ús: 51
(s) (s) Al
St, Aijo
Ar) (ot,
es,
11
(V*(s)
(st,, At)
(Bi BB) O((At)//2)
ds;
Σ (s Ad) (v*(s)
max pr
p(st,
At) A() +o (*((s)) +
Os: 18t
lo A A.) (s)
}
Al (((00()))32))))
Ös,Os
j=1
Simplifying the equation and taking the limit as At + O, we have the condition for the optimal value function:
V (st) 7
1
max Epp p(st, A.) +
Σ
(st, Ad)
(v)
Os; 18c
i=1
(s)
(s Ac)o A A.)
2
0s;0s;
A.3
Convergence of dTD
The existence and uniqueness of the weak solution V(-) to the linear parabolic problem 0,V TV V with Neumann boundary conditions follow from standard theory for uniformly parabolic equations.
A.3.1 Proof of Lemma
1
Proof.
We consider the linear elliptic boundary value problem
- (VV- = in S, n DVDVV = 0 on as,
where
()) V (V(s)VV(()) +bs)) VV(s), b:= n - d D.
Weak formulation. Let E HS) be test function. Multiplying the PDE by and integrating over 5 yields
/ VVo (CTV)== pv.
Using integration by parts for the divergence term and the Neumann condition n DVV on OS, we obtain
[ [nv +
2 [ (DVV) Vu - S (66-v))= pv.
Define the bilinear form B : H (S) + R and the linear functional f H (S) + by
B(u,v) v) = /
yuv /
2 (DVu) Vv - / Vu) v, f(v) | pv.
1
A weak solution is a function V H (S) such that B(V, v) f(v) for all H1'
14

---

Boundedness.
Since D E (S) and E L (S), there exists C > 0 such that for all v
(S)..
BBu, w) S -|||||==:|||| + 1 2 | Vellle + ||6] Vull
Moreover, since p (S)).. we have f(()) s | || $ |1 [oll hence bounded on H (S).
Coercivity,
Let U E (S). Using uniform ellipticity of D (i.e., 5 DE 2 al|5|13), we have
/ S (DV))
Ve
2 Vell2.
For the drift term, use Sauchy-Schwarn and Young's inequality: for any 0,
1S00 Voje |b||0 Voll S V
|661
12 4s
Choosing a/4 gives
1 (b.
Vo) (b.
S a Volli. + 4
|1b12.
|| 2.
a
Therefore,
B(c,0) = -110117. + 2 [ (DVu). Ve [ vojo
1
2 7 |bbl
|ull + a _ V
In particular, if y > |1b11.//a then there exists co > 0 such that B(v,v) 2 collellirss)) for all
E H (S), i.., B coercive on H (S).
Conclusion. Since (S) is a Hilbert space, B is bounded and coercive, and is bounded, the Lax-Milgram theorem implies that there exists unique weak solution //8 E (S) such that B(V",0 .)) = f(v) for all (S).
L
A.3.2 Proof of Proposition? 2
Proof. Let V (S) be the unique weak solution from Lemma and define e(t) == V - VT. Since V satisfies (7/ V= - and 3, - L + 7 p. we obtain the error evolution
0,e(t) = -- ())ct)) (c())
7
We use the weak formulation associated with the bilinear form B(--)) defined in the proof of Lemma Namely, for any test function H (S).
(die(t), u) 13(S) B(e(t),u).
Choosing e(l) yields the energy identity
1 d
2 di [()()))))s)))))))))))))))) = B(e(t),
By the coereivity estimate established in the proof of Lemma
we have for all w (S)
B(w, w) 2 (7-
|b/12-
||20 ||
b b:= = p div D.
α
15

---

The values chosen finally are Applying this with = e(t) gives
d [e(())BBss))) -
2 (7 | [ee(t)]
dt
7
Since y > ||612./a, Grönwall's inequality implies
(-2 7
2
(7 | )1) Ie((0)]
(e(()l))(s) S exp
and hence
((77 7
|101
[IV(1) - V (L)()) S <exp(-X0) (V(0) V V*
>
B
Implementation Details
B.1
Algorithm
The procedures for policy evaluation summarized in Algorithm
1
Algorithm 1 Policy evaluation with dTD
Input: policy
Output: V/
Initialize value function Vg with random parameter 0 for each training step do
Initialize buffer 'D = 0 and initial state
for each environment step do
~ (((()))
St+A1 ~ p( 1s,, ae) D - DU (st, aes Stt Pt)
end for
for each update step do
Sample a batch of random transitions from D
0 +-
ya +
p (Va(s
di
d.i )Vo(+)
di di
51 (5 86 2.5 (V
5t
pred, +-
At
Øs:
Af
Os,Uss 12
1=1
1=1==1
Update parameter 0 using gradient descent method 0 + argming 1/2 Σb Ed=1 (ya pred
end for end for
B.2 Efficient Computation of the dTD Loss
Since equation I0 involves the Hessian, it may that O(n²) (where n is the dimension of the observation space) computations are required. However, by rearranging the order of calculations, such using
(V((s) 2s2
0 Os
Ov
>l.)
Ast
As,
Ast.
Ast
Øs
we can avoid directly calculating the Hessian and achieve a computation complexity of (n)).
B.3 Hyperparameters for PPO
The search space of the hyperparameters is summarized in Table 2 summarized in Tables
16

---

Table 2: Hyperparameter search space
Hyperparameter
Search Space
environment steps per update
{8, 16, 32)
(number of parallel environment: 64)
number of epochs per update minibatch size learning rate
range(5. 20)
{256, 512]
log(interval 6, 5e 3))
normalize advantage gae lambda
(True, False]
interval(O.8, 0.9999)
clip range
interval(O.O, 0.9)
entropy coefficient
interval(O.0, 0.3)
value loss weight
interval(O.0, 1.0)
mixture raio |
interval(O.0,, 1.0)
Table 3: Best hyperparameters for PPO with TD and 3-dTD across environments
Hyperparameter
TD
3-dTD
Hopper
Halfcheetah
Ant
Humanoid
Hopper
Halfcheetah
Ant
Humanoid
environment steps/update
epochs/update
7
5
11
19
9
16
10
minibatch size
512
256
512
512
256
256
256
256
learning
1.18e-3
5,93e-4
3.71e-4
1.00-3
3.52e-4
3.29e-4
7.9de:5
1.08e-3
normalize advantage
False
False
False
False
False
False
False
False
GAE lambda
0.886
0.833
0.935
0.999
0.998
0.908
0.805
0.888
clip range
0.439
0.268
0.425
0.063
0.075
0.040
0.520
0.713
entropy coefficient
0.121
0.018
0.162
0.021
0.046
0.011
0.133
0.002
value loss weight
0.049
0.513
0.711
0.091
0.675
0.268
0.054
mixture ratio 3
0.572
0.742
0.241
0.332
B.4 Details of the A2C Implementation
B.4.1
Learning Curves
B.4.2
Hyperparameters for A2C
The search space of the hyperparameters is summarized in Table 4 summarized in Tables
The values chosen finally are
Table 4: Hyperparameter search space
Hyperparameter
Search Space
environment steps per update
{8, 16, 32}
(number of parallel environment: 64)
number of epochs per update minibatch size learning rate normalize advantage gae lambda entropy coefficient
range(5, 20)
(256, 512]
log(interval 6, 5e 3))
(True,False)
interval(0.8. 0.9999)
interval(O.0,, 0.3)
value loss weight mixture raio /
interval(O.0, 1.0)
interval(O.0,, 1.0)
B.5 Computing Infrastructure and Reproducibility
Computing infrastructure Experiments were conducted on machine with four NVIDIA Tesla V100 GPUs (32GB each) and an Intel Xeon E5-2699 CPU. Although all experiments can be executed on single GPU, multiple GPUs were used to run independent trials in parallel for efficiency.
17

---

-103
Hopper (noise: 0.00)
-10
Hopper (noise: 0.01)
-10
Hopper (noise: 0.05)
Episodic return
1.5
1.5
1.5
0.5
0.5
0.5
10
10
Halfchectah (noise: 0.05) 105
-10
Halfeheetah (noise: 0.00)
-10
Halfchectah (noise: 0.01)
-10
Episodic return
-10
-10
-10
-10
Ant (noise: 0.00)
-10
Ant (noise: 0.01)
-10
Ant (noise: 0.05)
Episodic retum
3
2
9
2
1106
0
2
100
0
Humanoid (noise: 0.05) -10
-10-
Humanoid (noise: 0.00)
-10
Humanoid (noise: 0.01)
110
Episodic retum
2
2
2
106
0
-00
0
2 Total episode step
-00
Total episode step
Total episode step
Figure 3: Performance of TD, and 3-dTD on continuous control benchmark. These results were obtained using the A2C algorithm. Each column corresponds to different noise levels (coef = 0.00, 0.01, 0.05), and each row corresponds to different environments. The tuned values for B-dTD were 0.24, 0.60, 075, 0.47 in Hopper, HalfCheetah, Ant, and Humanoid, respectively.
Table 5: Best hyperparameters for A2C with TD and B-dTD across environments
TD
8-dTD
Hyperparameter
Hopper
Halfcheetah
Ant
Humanoid
Happer
Halfcheetah
Ant
Humanoid
environment steps/update
32
16
16
16
8
16
epochs/update
7
5
11
16
9
12
minibutch size
512
256
512
512
512
256
256
512
learning rate
1.18e.3
5.93e-4
3.71e-4
1.40e-3
1 1.88e-6
3.52e-4
2.90e-6
6.83e-7
normalize advantage
False
False
False
False
False
False
False
True
GAE lambda entropy coefficient
0.886
0.833
0.935
0.999
0.890
0.950
0.960
0.827
0.121
0.018
0.162
0.021
0.031
5.38e-6
0.094
0.046
value loss weight mixture ratio 3
0.049
0.091
0.825
0.244
0.598
0.750
0.467
Training time Hyperparameter tuning typically took 6-9 hours depending on the environment. Training time for the final runs depended on the environment and ranged from 10 to 60 minutes.
Reproducibility All experiments were conducted with the random seed fixed in the training scripts. However, MuJoCo (accessed via Brax) uses its own internal random seed that is not directly controllable, so full determinism cannot be ensured. The code is available https for reproducibility.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We not only theoretically developed a model-free, TD method based on the HJB equation for continuous systems, but also validated the potential effectiveness empirically.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: On the theoretical aspect, we clearly noted that currently the proposed method is not applicable to off- aff-policy settings. On the empirical side, we found that the proposed method might work similarly the baseline method when the process noise is large.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
19

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The assumptions are clearly stated, and we added further explanations about them. Full proofs of the propositions provided in the main text and in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: All the experimental settings are described in the main text and the appendix. We also made the codes publicly available.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
20

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We made the codes publicly available. The RL environment used is originally publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
Codeeuomi ssinnpolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: All the experimental settings are described in the main text and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: The achieved rewards are compared considering the variance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
21

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
All the experimental settings are described in the main text and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer:
[Yes]
Justification:
We followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: The study strongly from the theoretical perspective, and we do not think is straightforward to discuss the direct societal impact at this stage.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
22

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We referred to the libraries we used in the experiment.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
23

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification:
We gave description along with the codes.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.MM policy (https: hateap///eerrpppc.cc///offeerere for what should or should not be described.
25


Reviews:
[{"review_id": "XwXUkbVpns", "reviewer": "Reviewer_q3JW", "summary": "In the present paper, the authors introduce differential temporal difference, a model-free approach to temporal differences in reinforcement learning. To derive the method the authors build on the Hamilton-Jacobi-Bellman (HJB) equation and use Ito's stochastic calculus to derive the algorithm for differential temporal differences from the HJB as well as a number of theoretical properties such as its contraction property. This approach is subsequently evaluated on 4 continuous control environments showing it outperforming the classical temporal differences approach.", "questions": "### Main Questions (mostly re the evaluation)\n- As mentioned above, the evaluation currently lack comparison to other algorithms to better position the work of the present paper in relation to the wider literature. In the introduction you point out a number of previous works, orthogonal to yours (line 37ff). Have you considered adding e.g. model-based approaches to TD to the evaluations to improve the paper's comparability to wider literature?\n- _\"It is compatible with on-policy methods such as A2C (Mnih et al., 2016) and PPO (Schulman\net al., 2017)\"_ (line 44f), why is this not expanded upon with dedicated evaluations? As is, this potentially very strong evaluation claim is not backed up by the evaluation design.\n- With the theoretical derivations of the new algorithm being one of the major strengths of the paper, how derived theoretical properties utilized, or even leveraged in the evaluations?\n\n### Minor Questions\n- You position the paper as mostly useful for applications in robot learning & autonomous driving, do you think that your approach could potentially also hold a lot of utility for applications in the sciences?\n- Is the DEHB method used for hyperparameter tuning truly the most efficient hyperparameter tuning approach for reinforcement learning? The 2023 study referenced here as source of this knowledge is outdated. I would recommend to remove this half-sentence in light of the significant body of work on hyperparameter optimization since then. Furthermore, I would argue that their findings are no longer broadly applicable, at the same time not making the hyperparameter optimizer not all of a sudden a bad or wrong hyperparameter optimizer to choose.", "limitations": "Limitations only discussed implicitly throughout the paper, not explicitly in a dedicated _\"Limitations\"_ section. It would aid the reader considerably to summarize the limitations in one succinct limitations paragraph.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The main strengths of the paper lie in its super high clarity of the writeup throughout the paper, complementing its strong theoretical derivations. The step-by-step derivations starting out from the Hamilton-Jacobi-Bellman equation using the Ito's stochastic calculus are very clear, and easy to follow. The derived theoretical results on the contraction are a clear result of this strong theoretical foundation to the newly introduced algorithm.\n\nThe weaknesses of the paper all lie in its evaluation, to hone in on a number of the weaknesses:\n- The evaluation as is show a lack of comparison to other algorithms, while utilizing standard libraries like Brax, whose interface to reinforcement learning frameworks has seen wide use. This is especially surprising given the author's acknowledgement of model-based counterparts to the proposed model-free approach of the paper, which would readily lend themselves as points of reference on the performance of the newly introduced algorithm.\n- The evaluations as is, only compare to a limited number of control environments, where a larger number of (continuous) control environments are readily available through the same software packages used by the authors. \n- Claims with regards to the utility of the new algorithm, such as its potential use in PPO & A2C algorithm, are not reinforced through the evaluation design.", "was_revised": true, "final_justification": "The authors' approaches main concerns raised during review such as the limited evaluations, and the limitations of some of the theoretical results have all been addressed during rebuttal. What remains is a worthy contribution to the field, which is wholly worthy of acceptance.", "rebuttal": "We thank the reviewer for the detailed and constructive feedback. We address the main concerns regarding the evaluation and clarify our design choices below.\n\n# Evaluation Updates and Justification\n## Evaluation on A2C and Model Oracle Settings\nWe originally considered PPO sufficient for our experiments, as it builds upon and improves A2C, and has been widely adopted as a de facto standard in RL research. However, we agree that the raw performance of the proposed dTD method can be more clearly assessed in A2C, where fewer confounding elements exist. We therefore conducted additional experiments using A2C, and we present only the final performance values in Table A1.\n\nWe additionally addressed the concern regarding comparison with model-based methods by introducing a model oracle setting in each environment, where the drift $\\mu$ and diffusion $\\sigma$ terms of the environment dynamics are assumed to be known. This setup serves as a reference point for evaluating the performance of our model-free approach. The corresponding final results are also included in Tables A1 and A2.\n\n## Comparison with Prior Work\nIn our view, a valid comparison baseline must meet both of the following:\n\n・ Estimate a Value or Q-function; \\\n・ Be compatible with deep RL frameworks.\n\\end{enumerate}\n\nAmong prior related studies, only Tallec et al. (2019) and Yıldız et al. (2021) satisfy the above two conditions (though they are applicable only to the deterministic setting). Although the overall approach of Tallec et al. (2019) is well-designed, its value function estimation component is essentially equivalent to classical TD, which we already included as a baseline.\nThe method of Yıldız et al. (2021) relies on accurate modeling of the environment dynamics. Our additional experiment with the model oracle includes such a configuration. As we used the true drift and diffusion terms, our evaluation is free of model estimation error.\n\nAs for Kim et al. (2021), their method focuses on improving the computation of the $\\arg\\max_a Q(s, a)$ operation, whereas our method modifies the TD update itself. It means that they target fundamentally different components of the RL algorithm. Comparing them is like comparing apples and oranges --- these are orthogonal contributions. In fact, our method could in principle be combined with HJ DQN, for example, by incorporating our TD update into Equation (22) of Kim et al. (2021).\n\nOther related studies mentioned in the paper are rooted in stochastic control theory, typically under limited settings such as finite-horizon or LQR formulations. These settings often require additional assumptions, such as time-dependent value functions $V(t, s_t)$ in the finite-horizon case, and are incompatible with modern deep RL pipelines and standard benchmark environments.\n\n## Number and Scope of Evaluation Environments\nThe aim of the work is to introduce a theoretically grounded model-free loss function for continuous RL and to show that HJB-based TD methods, previously considered incompatible with deep RL, can actually be implemented and run within standard deep RL pipelines.\nRather than focusing on achieving state-of-the-art results, our primary contribution is a conceptual one, particularly towards bridging stochastic optimal control and model-free RL.\nWe not only formulated the loss function but also demonstrated that it works with commonly known benchmarking tasks.\nWe therefore respectfully disagree with the concern regarding the number of evaluation environments.\n\nOur experiments are on four standard continuous control tasks, spanning a wide range of dimensionality from 10 to 244. We believe these environments are representative and sufficient to demonstrate the behavior of our method. The other gym tasks such as Pusher or Walker2d are unlikely to yield fundamentally new insights compared to the environments already included.\nWith that being said, we do acknowledge the value of including simple, low-dimensional environments to complement our analysis and provide additional clarity. To that end, we have added experiments on the Pendulum environment and included the results in Tables A1 and A2.\n\n# Answers to Remaining Questions\n## Empirical Reflection of Theoretical Insights\nThe contraction property derived in the theory implies that the convergence of our method is faster when the environment dynamics are smoother (i.e., have a smaller Lipschitz constant). Since our method targets the same objective function as classical TD, both approaches are expected to converge to similar value functions asymptotically. This theoretical insight is empirically reflected in the experimental results and discussed in the discussion section.\n\n## Applicability to Scientific Domains\nYes. In fact, recent work in diffusion models has applied stochastic control to enhance the aesthetic quality and prevent distorted contents [R1,R2]. These approaches typically require estimating the drift term of the reverse diffusion process, which in turn necessitates training an additional model solely for this purpose. Our method offers a theoretically grounded and model-free way to avoid this drift estimation step and may offer significant advantages in such applications.\n\n\n## Revised Wording Regarding DEHB\nWe agree with the reviewer. The current phrasing will be replaced with a more neutral statement to avoid overstatement, e.g.: \"We applied DEHB, a multifidelity optimization method that showed strong performance in prior work.\"\n\nWe hope that the additional evaluations and clarifications provided above help address the concerns.\n\n\n**TableA1: Final episodic return of A2C:** Each cell shows the final episodic returns under three different noise levels: 0.00 / 0.01 / 0.05.\n|                | Pendulum         | Hopper           | HalfCheetah       | Ant               | Humanoid          |\n|----------------|------------------|------------------|-------------------|-------------------|-------------------|\n| TD             | 1000 / 1000 / 1000  | 1818 / 1846 / 1587  | 8793 / 9273 / 7904   | 1964 / 2182 / 2007   | 285 / 289 / 287   |\n| β-dTD              | 1000 / 1000 / 1000  | 2645 / 2565 / 2436  | 11948 / 10494 / 8919   | 2569 / 2469 / 2255   | 301 / 320 / 317   |\n| model-based        | 1000 / 1000 / 1000  | 2864 / 2834 / 2607   | 12182 / 12696 / 10537  | 2716 / 2892 / 2605   | 320 / 331 / 323   |\n\n\n\n**Table A2:  Final episodic return of PPO:**  \nEach cell shows the final episodic returns under three different noise levels: 0.00 / 0.01 / 0.05.\n\n|         | Pendulum         | Hopper           | HalfCheetah       | Ant               | Humanoid          |\n|---------|------------------|------------------|-------------------|-------------------|-------------------|\n| model-based   | 1000 / 1000 / 1000  | 3952 / 3904 / 3707  | 20102 / 18738 / 13285   | 4202 / 4037 / 3796   | 430 / 418 / 411   |\n\n\n[R1] Masatoshi Uehara et al.(2024) Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Review. arXiv:2407.13734, 2024.\n\n[R2] Wenpin Tang.(2024) Fine-tuning of diffusion models via stochastic control: entropy regularization and beyond. arXiv:2403.06279, 2024."}, {"review_id": "2dW4y3vVSL", "reviewer": "Reviewer_jPHL", "summary": "This paper proposes an alternative algorithm to TD learning, called differential TD (dTD), for continuous-time, stochastic RL problems. Contrary to existing approaches from the optimal control literature, this algorithm is model-free (sample-based) like TD learning, but exploits the continuity of the value function. The authors demonstrate the practical potential of this approach, when mixed with classical TD iterations.", "questions": "- The assumptions of Proposition 2 are appropriately discussed in the paper. It is indeed known that $V$ has in general little regularity in the deterministic case. However I was wondering if something more could be said in the stochastic setting: the noise in the dynamics is usually seen as having a regularizing effect on the value function. See for example [Fleming and Rishel, *Deterministic and Stochastic Optimal Control*, 2012], where the Brownian motion makes the value function at least C^2 in the space variable.\n- In Eq. 8, a term $O(\\Delta t^{3/2})$ is missing from the rhs of the equation.\n- Is it possible to exhibit a toy example where exploiting the continuity of the value function is key, and makes dTD perform significantly better than plain TD? Did the experiments that you led give you any insight on the type of problems where dTD is expected to perform best? \n- I understand that mixing dTD with TD is better in terms of stability, however I believe that dTD (with $\\beta=1$) should be reported as well in the plots, even as a baseline. \n- Can you provide some more details on the parametric representation of $V$? Is it exactly the same for the TD, dTD and naive dTD algorithms? Is it adapted to the different problem instances?\n- Can you please clarify the exact difference between dTD and naive dTD, and why their performance is so different?", "limitations": "Limitations concerning stability issues and the large discount factor regime are appropriately mentioned in the paper.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "This paper, well-written and theoretically-grounded, tackles an interesting problem, in the relatively under-studied continuous setting in RL. The proposed algorithm is sample-based, hence compatible with RL problems. While it is limited to fixed-policy TD-based approaches, the paper shows with PPO how it can be applied to optimize the policy as well. The experiments are thoroughly led, with interesting discussions on the influence of the different parameters. The limitations are mainly related to the experimental comparison with the plain TD algorithm, where the actual advantage of dTD is difficult to assess in all situations. Here are more detailed limitations (see also questions below):\n- While the contraction analysis provided in Proposition 2 is important, a consistency result as $\\Delta t \\rightarrow 0$ is missing from the analysis. When dTD is applied with a finite $\\Delta t$, can we say anything about the error on the value function or the Bellman error?\n- In the experiments, the naive dTD and dTD algorithms have very different behaviors. However, since the  loss function is mentioned to be the MSE, I don’t understand how changing the target and prediction would change anything (up to a scaling factor of $\\gamma$)? This is the most confusing detail of the paper, and I think the precise difference between the two updates should be made much clearer.\n- TD is an algorithm designed for policy evaluation. Yet in the experiments, only PPO is tested, while TD is only a sub-element of it. Is it possible to assess the performance of dTD on policy evaluation only, where the distance to $V^{\\pi}$ can be directly evaluated? Maybe the influence of the different parameters would be easier to judge on this specific sub-problem.\n- As discussed in the paper, the optimal $\\beta$ depends on the problem. However, it is quite difficult to judge whether finding that some $\\beta >0$ performs better than $\\beta=0$ on some problems is indeed significant, or is only due to the fact that we are optimizing one more free parameter.", "was_revised": true, "final_justification": "The authors have answered most of the my questions. While certain weaknesses may remain in the paper, I believe it is of interest for the community, and could pave the way for future interesting work. Therefore I update my score from 4 to 5 and recommend to accept this paper.", "rebuttal": "Thank you for your thoughtful and insightful questions. Below, we address them in detail:\n\n# Response to Limitations Raised by the Reviewer\n## On the analysis of finite difference\nAs shown in Equation (7), the finite-difference approximation $\\mathbb{E}\\left[\\frac{S_{t+\\Delta t} - S_t}{\\Delta t}\\right]$ \ndeviates from the true drift term $\\mu$ by an error of order $O(\\Delta t^{3/2}).$ This deviation directly affects the contraction coefficient in Proposition 2, as it introduces a corresponding perturbation to the Lipschitz constant of the dynamics.\n\nAs a side note, in practice, the algorithm uses the empirical finite difference \n$\\frac{S_{t+\\Delta t} - S_t}{\\Delta t}$ instead of its expectation. This can be decomposed as:\n$$\n\\frac{S_{t+\\Delta t} - s_t}{\\Delta t} = \\mathbb{E}\\left[\\frac{S_{t+\\Delta t} - s_t}{\\Delta t}\\right] + \\left(\\frac{S_{t+\\Delta t} - s_t}{\\Delta t} - \\mathbb{E}\\left[\\frac{S_{t+\\Delta t} - s_t}{\\Delta t}\\right]\\right).\n$$\nThe second term in this decomposition forms a martingale difference noise. Under standard Robbins-Monro step-size conditions, such noise vanishes asymptotically, ensuring that it does not affect the long-term consistency of the learning dynamics.\n\n## On the difference between naive-dTD and dTD\nIn the formulation of the loss function, the target term is detached from the computation graph (i.e., no gradient flows through it), while the prediction term is backpropagated through; please refer to Section 5.1 and Table 1 of the paper.\nAs noted there, this design choice follows the standard practice in deep RL, where MSE is minimized between a fixed target and a learnable prediction.\nFor the Bellman equation, the target (right-hand side of the Bellman eq.) includes the true reward $r_t$ and is generally considered more reliable than the prediction (left-hand side), which motivates treating it as fixed during training.\n\nHowever, the HJB equation is not merely an Itô expansion of the Bellman equation; it involves algebraic rearrangements, including swapping the left- and right-hand sides. As a result, it is not a priori clear which side is more reliable and should be treated as the target term. The choice of parameterization is therefore nontrivial. Although the naive-dTD formulation appears more conventional in structure, our proposed dTD method, which treats the value function as the trusted quantity and adjusts gradients accordingly, yields better empirical results. This behavior can be understood from the fact that the HJB equation ultimately models the evolution of the value function itself, not its gradient.\n\n## On the direct evaluation of TD\nValue-based reinforcement learning generally follows either of two approaches: value iteration or policy iteration. As discussed in the paper, value iteration is not compatible with our formulation. In policy iteration, one alternates between policy evaluation and policy improvement. Natural extensions in this context include SARSA. While SARSA is the simplest form of policy iteration, it is incompatible with our continuous action setting. Although policy evaluation alone cannot be used as a standalone RL algorithm, it can be meaningfully incorporated into broader frameworks such as actor-critic methods. Among actor-critic variants, A2C is the most straightforward instantiation, but we chose PPO for its practicality and widespread use in deep RL today.\n\nWith that being said, we fully agree with the reviewer's point that the effect of dTD would be more clearly isolated in simpler settings. For this reason, we conducted additional experiments using A2C, which are included in the table below.\n\n# Answers to Questions\n## The assumptions of Proposition 2\nThe solution to the HJB equation is known to be a viscosity solution, which is continuous but not necessarily everywhere differentiable. However, the differential operations required by the contraction analysis are applied to the function approximator, not the true value function. Since neural networks are almost everywhere twice differentiable and capable of approximating any continuous functions including the viscosity solutions. We believe this does not pose a practical issue.\n\n## Toy Example\nWe added the Pendulum task as a simple toy example (see Table A1 and A2).\n\n## dTD with β=1\nWe have added experiments for the case $\\beta = 1$ to directly address this point. The results are included in Table X and support our observation that combining TD and dTD improves stability and performance across different parameter choices (see Table A1 and A2).\n\n##  parametric representation of $V$\nAll methods use the same three-layer fully connected neural network. The full implementation details are specified in the paper, and the code is available at the provided link.\n\n## Additional Info\nWe only mentioned the large discount factor regime (i.e., small discount factor in discrete time) as a possible stabilizing factor. As stated in Appendix B.2, the discount values used (0.8–0.99) are within a typical range in RL.\n\n**TableA1: Final episodic return of A2C:** Each cell shows the final episodic returns under three different noise levels: 0.00 / 0.01 / 0.05.\n|                | Pendulum         | Hopper           | HalfCheetah       | Ant               | Humanoid          |\n|----------------|------------------|------------------|-------------------|-------------------|-------------------|\n| TD             | 1000 / 1000 / 1000  | 1818 / 1846 / 1587  | 8793 / 9273 / 7904   | 1964 / 2182 / 2007   | 285 / 289 / 287   |\n| β-dTD              | 1000 / 1000 / 1000  | 2645 / 2565 / 2436  | 11948 / 10494 / 8919   | 2569 / 2469 / 2255   | 301 / 320 / 317   |\n| model-based        | 1000 / 1000 / 1000  | 2864 / 2834 / 2607   | 12182 / 12696 / 10537  | 2716 / 2892 / 2605   | 320 / 331 / 323   |\n\n\n\n**Table A2:  Final episodic return of PPO:**  \nEach cell shows the final episodic returns under three different noise levels: 0.00 / 0.01 / 0.05.\n\n|         | Pendulum         | Hopper           | HalfCheetah       | Ant               | Humanoid          |\n|---------|------------------|------------------|-------------------|-------------------|-------------------|\n| model-based   | 1000 / 1000 / 1000  | 3952 / 3904 / 3707  | 20102 / 18738 / 13285   | 4202 / 4037 / 3796   | 430 / 418 / 411   |"}, {"review_id": "OaM2numoM5", "reviewer": "Reviewer_TdDL", "summary": "This paper introduces a Reinforcement Learning algorithm that updates the value network using the stochastic HJB equation, thus bridging discrete-time and continuous-time insights. This method is tested on the MuJoCo environments and outperforms PPO.\n\nUnfortunately, although the experimental results are promising, I am concerned about the author's familiarity with HJB equations and important characteristics of the problem. I am willing to increase my score if the authors focus on making their work more grounded in the literature on viscosity solutions and continuous-time policy iteration in their rebuttal.", "questions": "* In the abstract, \"Bridging stochastic optimal control and model-free reinforcement learning\", what is the difference?\n* In Appendix A.3, where you use Ito's lemma, you are missing $\\Delta t$ in some parts of the equation.\n* How is Proposition 2 used in the numerical experiments? How are $L_v$ and $L_\\sigma$ computed?", "limitations": "* I appreciate that the authors acknowledge some limitations (e.g., instability, off-policy inapplicability, large $\\gamma$) of their work.\n* As mentioned above, I suggest that the authors conduct an exploration of regions where $V$ isn't first or second-order differentiable.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n* The paper is well-written and mathematically easy to follow. I followed the proof of Proposition 1 and the details on the HJB equation and believe both are correct. Appendix A.1 is correct and carefully grounded in the literature on stochastic differential equations. \n* Although Proposition 2 is based on problematic assumptions, I went through the proof and believe it is correct.\n* The derivation from the Bellman equation to the HJB to the dTD update is clear and well-motivated.\n* The experimental results seem promising. PPO with $\\beta$-TD significantly outperforms PPO in six out of twelve environments and shows comparable performance in the remaining environments (except half-cheetah)\n\n**Weaknesses**\n\n* I find Proposition 2 problematic. How are the assumptions verified? Assumption (i) is strong and may not hold in general. This undermines the generality of the contraction proof. $V$ will be, at best, locally Lipschitz in some sub-domains. Furthermore, it is not scalable: as n and m increase, $\\gamma$ needs to increase as well, which can severely limit the model's ability to consider long-horizon planning. How are $L_v$ and $L_\\sigma$ computed in numerical experiments?\n* In general, the value function will not have continuous first or second order derivatives - rather, it will be a viscosity solution (indeed, the study of viscosity solutions was motivated by the fact that the value function is not generally differentiable). For example, in LQR settings, the value function is not differentiable at the origin. I highly recommend that the authors read the paper [1] on neural network-based methods of mitigating these issues.\n* Fundamentally, the proposed method combines discrete-time and continuous-time solutions, which I believe is problematic. The proposed algorithm is based on PPO and thus uses the policy gradient (PG) theorem [2], which is derived in the discrete-time domain. If the authors want to use a policy gradient algorithm, they need to build their method based on the continuous-time variant of PG, for instance, pathwise policy gradient estimators [3].\n* Important baselines are missing: (1) Soft Actor-Critic [4], (2) HJ DQN [5].\n\n[1] Yiming Meng et al. \"Physics-Informed Neural Network Policy Iteration: Algorithms, Convergence, and Verification.\" ICML 2024.\n\n[2] Richard Sutton et al. \"Policy gradient methods for reinforcement learning with function approximation.\" NeurIPS 1999.\n\n[3] Remi Munos. \"Policy gradient in continuous time.\" JMLR (2006).\n\n[4] Tuomas Haarnoja. \"Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor.\" ICML 2018.\n\n[5] Jeongho Kim et al. \"Hamilton–Jacobi Deep Q-Learning for Deterministic Continuous-Time Systems with Lipschitz Continuous Controls.\" JMLR (2021)", "was_revised": true, "final_justification": "Through the rebuttal, the authors have addressed my concern with the theoretical grounding of their paper, namely that the value function is not Lipschitz and smooth. This assumption has been relaxed, and the condition for contraction of the HJB operator has been replaced with an inequality that is grounded in more realistic assumptions, thus significantly improving the mathematical foundation of the paper.", "rebuttal": "We thank the reviewer for their detailed feedback and for engaging with our work.\nDespite the favorable evaluation regarding the quality (good), clarity (excellent), significance (good), and originality (good) of the paper, seemingly some miscommunication and misunderstanding led to the current overall rating.\nWe believe the answers below would be a thorough clarification.\n\n# Response to Limitations Raised by the Reviewer\n## How to compute Lipschitz coefficient of V?\nAs shown in Appendix B.3, we do not need to compute the Lipschitz coefficient of $V$. This requirement appears only in the theoretical analysis to ensure convergence. We believe this limitation is acceptable, as many widely adopted methods, such as SGD or Nesterov's accelerated methods (including variants like Adagrad), make similar assumptions solely for theoretical purposes.\nThere, as is the case with our work, we do not need to know the Lipschitz constant in practice.\nFor example, typically one does not determine the step size of SGD by explicitly computing the Lipschitz constant of the objective function.\n\n## Differentiability of $V$\nAs the reviewer pointed out, the viscosity solution of the HJB equation is continuous but not necessarily everywhere differentiable. However, the differentiability requirement in our analysis applies not to the viscosity solution itself, but to the function approximator $V_\\theta$. Neural networks can be designed to be twice differentiable and can approximate any continuous function including viscosity solutions.\n\n## Continuous time policy gradient\nWe do not combine discrete-time and continuous-time formulations in the inconsistent way that the reviewer seems to have in mind.\nWe guess some misinterpretation of the policy gradient theorem resulted in the reviewer's concern.\nThe policy gradient theorem was originally introduced in discrete-time RL, but the derivation is not specific to discrete time. The core idea relies on the log-derivative trick:\n$$\n\\begin{aligned}\\nabla_\\theta \\mathbb E_{x\\sim P_{\\theta}(x)}[f(x)] &= \\nabla_\\theta\\int f(x)P_\\theta(dx) \\\\ &=\\int f(x)\\nabla_\\theta P_\\theta(dx) \\\\ &=\\int f(x)\\frac{\\nabla_\\theta P_\\theta(x)}{P_\\theta(x)}P_\\theta(dx) \\\\ &= \\mathbb E_{x\\sim P_\\theta(x)}\\left[f(x)\\frac{\\nabla_\\theta P_\\theta(x)}{P_\\theta(x)}\\right] \\\\ &= \\mathbb E_{x\\sim P_\\theta(x)}\\left[f(x)\\nabla_\\theta\\log{P_\\theta(x)}\\right] \\\\ \\end{aligned}\n$$\nThis formulation applies equally in both discrete and continuous-time settings. In discrete RL, we take $f \\rightarrow \\sum_{t=0}^{\\infty} \\gamma^t r(s_t, a_t)$; in continuous RL, we use $f \\rightarrow \\int_{0}^{\\infty} e^{-\\gamma t} \\rho(s_t, a_t) dt$. When applying the method to MuJoCo environments, which operate in discrete simulation steps, we discretize the continuous-time formulation accordingly. This discretization ultimately yields the same form as standard discrete-time policy gradient methods.\n\nThe cited paper by Munos [3] proposes a method for improving policy gradients in continuous time, but it is not a formulation of the baseline continuous time policy gradient itself.\n\n## Missing baselines\nSoft Actor-Critic is an off-policy method, and as noted several times in the paper, our proposed approach is designed for on-policy algorithms such as PPO and A2C.\n\nDirect comparison to HJ DQN [5] is very meaningful. It focuses on improving the computation of the $\\arg\\max_a Q(s, a)$ operation, while our method directly modifies the TD update itself. These approaches target fundamentally different components of the RL algorithm, and comparing them is like comparing apples and oranges. In fact, our method could in principle be combined with HJ DQN—for example, by incorporating our proposed TD update into Equation (22) of [5]. The two are not competing methods but rather orthogonal in design.\n\n# Answer to questions\n## Difference between stochastic control and RL\nIn stochastic control, the HJB equation is typically solved analytically under various assumptions. In contrast, RL solves the Bellman equation in a data-driven manner without such assumptions. In continuous RL, there has been interest in solving the HJB equation in a data-driven manner similar to RL. However, existing approaches typically require either access to the true dynamics or an explicit model estimation step, creating a gap from how RL is practiced. In this work, we demonstrate that it is indeed possible to solve stochastic control problems in a fully model-free, RL-style manner.\n\nWe have also included experimental results for A2C and the model-known setting as part of our additional analysis. For further details, we kindly refer you to our responses to the other reviewers."}]

Author rebuttal:

Meta-review: This paper introduces differential TD learning, derived from the HJB equation. Reviewers appreciated the rigor of the derivations and the potential to bridge stochastic control with RL. The main weakness is the limited empirical validation, with missing baselines and restricted scope of experiments. Still, the methodological contribution is solid and judged valuable for the community. The decision is accept.
Decision: Accept (poster)