Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Control
Authors: Chengxiu HUA
Jiawen Gu
Yushun Tang
Venue: NeurIPS 2025 poster ()
Keywords: Stochastic Control; Reinforcement Learning; Diffusion Policy; Continuous Q-Learning; Martingale Orthogonality Condition
Primary area: reinforcement_learning
TL;DR: We introduced a Q-function framework for continuous-time reinforcement learning and develope the Continuous Q-Score Matching (CQSM) algorithm.

Abstract: Reinforcement learning (RL) has achieved significant success across a wide range of domains, however, most existing methods are formulated in discrete time. In this work, we introduce a novel RL method for continuous-time control, where stochastic differential equations govern state-action dynamics. Departing from traditional value function-based approaches, our key contribution is the characterization of continuous-time Q-functions via a martingale condition and the linking of diffusion policy scores to the action gradient of a learned continuous Q-function by the dynamic programming principle. This insight motivates Continuous Q-Score Matching (CQSM), a score-based policy improvement algorithm. Notably, our method addresses a long-standing challenge in continuous-time RL: preserving the action-evaluation capability of Q-functions without relying on time discretization. We further provide theoretical closed-form solutions for linear-quadratic (LQ) control problems within our framework. Numerical results in simulated environments demonstrate the effectiveness of our proposed method and compare it to popular baselines.

Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Control
Chengxiu Hua!, Jiawen Gu Yushun Tang
'Southern University of Science and Technology, HHuawei Technologies Co., Ltd.
12331005@mai1. sustech. cersstech.edu..oom en, jwgu. hkuRoutlook. Sangys2022mmai.sauuteeee sustech. eeutteechedd..c
Abstract
Reinforcement learning (RL) has achieved significant success across a wide range of domains, however, most existing methods are formulated in discrete time. In this work, we introduce a novel RL method for continuous stochastic differential equations govern state-action dynamics. Departing from traditional value function-basee approaches, our key contribution is the characteri- zation of continuous-timg time Q-functions via martingale condition and the linking of diffusion policy scores to the action gradient of a learned continuous Q-function by the dynamic programming principle. This insight motivates Continuous Q-Score Matching (CQSM), a score-based policy improvement algorithm. Notably, our method addresses long-standing challenge in continuous-time RL: preserving the oction-evaluation capability of Q-functions without relying on time discretization. We further provide theoretical closed-form solutions for linear-quadratic (LQQ trol problems within our framework. Numerical results in simulated environments demonstrate the effectiveness of our proposed method and compare it to popular baselines.
control, where
Introduction
RL has achieved substantial success across a wide range of domains over the past decade 44 Most existing approaches adopt discrete-tine formulation, typically modeled as Markov Decision Process (MDP) [3, [6]. where agents interact with the environment at fixed time intervals. However, many real-world systems- such autonomous driving in dynamic traffic conditions 477 robotic manipulation 34 and high-frequency algorithmic trading [27] exhibit continuous, fine- fine-grained dy- namies that are inadequately captured by discrete-time models. These applications naturally motivate the need for continuous-time reinforcement learning (CT-RL) frameworks that more faithfully repre- sent the temporal structure of decision- making. Recent works on CT-RI. have explored stochastic exploration techniques [15), and model-free learning methods for diffusion generative models fine-tuning (58 [7 and financial applications [19 4
modeling using stochastic differential equations (SDEs) [I] 211, entropy-regulant
Despite these advances, value-based methods like Q-learning
cornerstone of discrete discrete-timme
RL remain challenging to adapt to the continuous time setting. Traditional Q-learning algorithms (e.g.. SARSA [44, DQN (32) rely on estimating state-action value functions via temporal-differene
(TD) learning and have demonstrated strong performance in discrete action spaces. There is line of work on discretizing continuous action spaces to apply Q-learning in high-dimensional continuous control settings [49 40 20). Discretizing continuous actions is common approach to extend Q-learning, but it often struggles with scalability in high-dimensional spaces and relies on
Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

discrete- timetet-timeeee assumptions. However, extending Q-learning to continuous action spaces introduces major challenges. Early work [33 proposed neural betwork-basee formulations for continuous Q-learning, but later studies B reported severe performance drops in high-dimensional settings due to the curse of dimensionality. Moreover, when Q-learning is directly extended to continuous time, the Q-function tends to collapse into an oction-independent value function [46], losing its ability to distinguish between actions-a critical property for decision -maiin... In [25), they show that the discrete Q-learning algorithm is noisier and slower in convergence speed compared with their proposed continuous PG and little q learning algorithms.
To bridge this gap, recent work has explored Q-function-barn based frameworks that preserve action dependence in continuous time. For example, [1O) replaced the Q-function with generalized Hamiltonian, while [25] proposed little q-learning, a sume-discretization--e
method based on first-order Q-function approximations, which achieved faster convergence than discrete-time soft Q-learning (SARSA). Score-matching-ee
methods [35] have also emerged as promising tools for learning diffusion policies, though they still rely on time discretization. Other approaches, such as (26). incorporate the action state variable by constraining the action process to be absolutely continuous with bounded growth. However, these methods are limited to deterministic dynamics and require upfront discretization of the continuous-time problem. These constraints underscore key open challenge: how to design a principled and scalable Q-learning framework for stochastic, continuous-time environments that maintains the action characteristies of the Q-function. We address this by introducing a novel Q-function formulation and a corresponding algorithm, Continuous Q-Score Matching (CQSM), which operates directly in continuous time and supports stochastic dynamics.
Major contributions. This work makes the following key contributions: (1) Continuous-Time Q-Function Characterization. We derive Bellman equation (also known as the Feynman-Kac formula) for continuous-time Q-functions. This bridges discrete Q-learning with continuous control theory. We rigorously characterize Q-functions for a given score function using martingale condition defined over an enlarged filtration that incorporates both state and action noise. Based on this foundation, we propose new algorithms for direct Q-function learning, analogous to policy evaluation and policy gradient methods in 223 244.
(2) Score Improvement Theorem and CQSM Algorithm. We establish a score improvement theorem by the dynamic programming principle that enables principled policy updates in continuous time. By coupling the Q-function with the score function of a diffusion policy, we develop model- free, aetor-eritic-style algorithm: Continuous Q-Score Matching (CQSM). This algorithm facilitates efficient policy improvement using only the denoising score function.
(3) Analytical Validation via LQ Control. We resolve the LQ problem under measurable scores, demonstrating the theoretical soundness of our framework. LQ problems are fundamental in control theory, they serve as tractable approximations to more complex nonlinear systems, with practical relevance in areas such as algorithmic trading (2 and resource management [12). Using analytical solutions, we compare CQSM to policy gradient and little q-learning methods, which are the action- independent value-based RL models, highlighting both its theoretical guarantees and numerical advantages.
In Section 2 we review related work relevant to our method. Section ( introduces the continuous- time reinforcement learning formulation using stochastic differential equations and presents key preliminary results. In Section 4 we develop the Q-learning theory in continuous time, establishing martingale characterizations. We further extend the analysis to the mnfinitehhorizon setting and prove the score improvement theorem. Section 5 presents numerical evaluations on LQ control tasks, comparing CQSM with policy gradient and little q-leanning methods. Finally, Section concludes with summary and discussion of future directions.
2 Related Works
In this section, we review existing work across four areas related to our method: stochastic optimal control, continuous reinforcement learning, diffusion Q-learning, and behavior cloning.
Stochastic Optimal Control. Our approach builds on classical stochastic optimal control theory 57 particularly through the development of the Hamilton-JcooiiBe -elllma (HJB) equation for

---

continuous-time Q-functions. While traditional stochastic control frameworks often assume full knowledge of the system dynamics B3, we instead assume model for the dynamics of state-action pairs. This shift motivates new theoretical developments that underpin our Q-learning framework.
Continuous Reinforcement Learning. The formulation of CT-RL in stochastic settings- where state evolution follows stochastic differential equation (SDE) dates back to B3 7. though early work lacked data-driven learning mechanisms. Recent advances have introduced more practical formulations. For instance, [52] proposed an exploratory control framework for continuous RI,, while (53] [23), and [24] extended this line of work to mean-variance objectives, policy evaluation, and policy gradients, respectively. (25 further introduced the notion of little q-value, leading to continuous analogue of Q-learning. Additional developments include mean-field RL with continuous dynamics [28]. jump-diffusion extensions [14], and aefinite-horizon variants of TRPO and PPO [59]. Building on these foundations, we advance the study of continuous-úme Q-learning under diffusion policies.
Diffusion Q-Leearning Diffusion Q-learning [55] integrates diffusion models with Q-learning by using Q-values as training objectives and backpropagates through the diffusion model. More recently, [5] proposed model-free online RL method based diffusion policies. (35] further established a connection between diffusion-based policies and the Q-function by relating the policy score the action gradient of the Q-function. Building on this line of work, [54] employed entropy estimation to balance exploration and exploitation in diffusion policies, improving the performance of the policy. In parallel, (30 generalized diffusion model training by reweighting the conventional denoising score matching loss, leading to two efficient algorithms for training diffusion policies in online RL without requiring samples from optimal policies. However, their approach relies on time discretization and requires injecting noise into the final action of the diffusion chain. In contrast, our work preserves the full action-evaluation capability of the Q-function in continuous time, without relying on any discretization in either time or action space. Our method is grounded in martingale- based formulation of the HJB equation, which provides principled theoretical foundation for continuous-time Q-learning.
Behavior Cloning. Behavior cloning focuses on imitating expert trajectories without access to reward signals. Diffusion models particularly well-suited for this task due to their generative flexibility and natural alignment with score-matching objectives. Recent works [2 37 have applied diffusion models to behavior cloning by framing policy learning as distribution-mateed over expert data. These approaches inspired our incorporation of score-matching terms into the objective. However, our framework goes beyond imitation, enabling policy improvement through Q-function learning in continuous time.
problem
3 Formulation and Preliminaries
In this section, we introduce the continuous-tine time RL formulation using stochastic differential equa- tions and present key preliminary results.
Notation. We introduce the non-standard notation used throughout the main text and appendix. For vector x, denote by ||ж||| the Euclidean norm of æ. For function f on Euclidean space, V (resp. V²f) denotes the gradient (resp. the Hessian) of f. The Bellback-Leibler (KL) divergence of two positive density functions f.g is defined as Dxu(fl) = SA log [(o) (() da. Define an operator L : 2 (FT x Rd) n C(R" R x Rd) + C(R) [57] associated with the diffusion process as:
Lip(x, a) = V Vap((z a) bx bx + V (ap(z, V + 7 (oxo + 2 [Voo,V2
In both the main text and the proofs, we refer frequently to the score of the action distribution, denoted by V. This vector field defines the temporal evolution of actions and serves as a proxy for the true score V, log π (a/æ) where π refers to the action distribution.
Continuous RL. Let d, n be positive integers, T O. We denote the state as X, E R and the action 0r dd with t [0, T). We consider the following stochastic, continuous-tine time setting for
3

---

state and action dynamics:
(I) where [0, R" x R + R corresponds to the score of our policy, which serves as the primary optimization variable in this setting, bx, [0, continuous state and action dynamics, and ox Fa are functions from [0, T] R" R to positive semidefinite matrices in Ruxn and IRd respectively. The processes driven by independent Brownian motions: B {B.x 0} and Ba {B= 2 0}. All processes are defined on a filtered probability space FP, (Fs
dX, = bx X,, ar) dt + ox,t, az) d da, = (t, X1, a,) dt + talt, d')
Rd TRd > IR corresponds to the
=20) where (Fa)s20 the natural filtration generated by standard n-dimensional Brownian motion Bx and standard d-dimensional Brownian motion B The (continuous-tima) Q-function under any given is defined as
1 = EP f. ((ss.Xs,.)) -2 X ds h(Xr, ar) X, 2, ar =
(2)
where Eº is the expectation with respect to both Brownian motions B, and B r: [0, R x Rd - IK and h Rn Rd - R are running and lump-sum reward function, respectively, and À is the regularization coefficient governing the cost of large score magnitudes. The goal is to find an optimal score function V/* E II where II denotes the set of admissible diffusion scores, such that the optimal Q-function
Q x a) sup Q 2, a; V.
well
We now give precise definition of the admissible score set IT.
Definition 1. A score Ti is called admissible if
(i) V
T (t, X2, az) : 0} is adapted;
(ii) EF (s, Xx, a. a,) ds cc.
Details regarding the poss-desssssess of the control problem [ are provided in Appendix A
The score matching term 1/2 \ / a. 112 in the objective can be interpreted from the following two perspectives:
Quadratic Execution Costs. Let a denote the investor's portfolio position. The score matching term captures the quadratic costs of execution trades of size dt, where À quantifies the level of transaction costs. This is consistent with execution cost models in portfolio optimization M1:
Policy Regularization via KL Divergence. Suppose the diffusion coefficient aa is deterministic and X1, ax) ) ba (1, Xr, 0e) + utt, ae (t) where represents control. Under this setup, the score-matching cost admits an interpretation as a KL divergence between trajectory distributions. Specifically, let π base (ar [r, a) denote the distribution over terminal actions when = and let n" (ar Ir, 0.) denote the distribution under control u. By the Section 3 in (6). we have
DKL(T" (ku(z"(arlz; (ar|z, (000 a)) = EE? [ |u(s, Xx, az) 12 ds X; = x, a, =
(4)
Setting = (1) the original score-matching term aligns exactly with this KL regularization. The corresponding objective can then be interpreted as:
Q(1,2,0) = sup E² aell
r(X., as) ds - M(Xx( ar) X, = ar
(5)
- -DD (eKz(==(r (ar((rr Ir, a)l b (ar(z, a)).
base
Thus, the KI. term encourages the optimal policy to remain close to the base dynamics, introducing form of regularized policy improvement.
4 Continuous Q-Score Matching Algorithm
In this section, we develop a continuous-time Q-learning framework using martingale characteriza- tion and the HJB equation. This offers an alternative approach to policy improvement.

---

4.1 Dynamic Programming and HJB Equation for Q-Function
By the dynamic programming principle, the Q-function satisfies the following HJB equation:
EQ(t, aQ ot at, r(t,,,,) 21, - = 0.
sup FEll
(6)
Note that the terms V.Q -.QQ-b(ttaa 2, VaQ (,, b, a), jir (axa fur (o [2QQ and
r(t, 2, a) are all independent of Hence, the supremum in (6) is attained (t, x, a) V. 21, a). Substituting this back into the HJB equation gives the following nonlinear partial differential equation characterizing the optimal Q-function:
8Q Ot (t, + +r((,,,,)))))))) V.Q(t)), bx(t, 2 2 1V.
+22 (ox((t.z,)oxx(ttzpoo V + 1tt (0 (V = 0, Q (()
(7)
We now focus on the optimal Q-function associated with the optimal score 1r*, To avoid unduly technicalities, assume throughout this paper that the Q-function Q E ((0,.22(0, T) R IRd) C) x R x Rd)) satisfies the polynomial growth condition in the joint state-action variable 2 = (x, a). The following theorem establishes the key martingale characterization underpinning
policy evaluation for diffusion- fiffussio-bbase policies.
Theorem 1. If ( V is the Q-function associated with the score V if and only if it satisfies
terminal condition Q(T, x, a; v = -1lx, a), and for all (x,aa E Rn x Rd, the following process
M. Q(s, X., as: V) + J." [[u,,Xus an) _ (1, X., aw)l
du
(8)
(1F.) $20. P)-matiingale on [t, T. Conversely, if there is continuous Q-function Q such that for all (xr E R." Rdd, M2 a martingale, where
J.¹ rr(, Xu,
1 All Xu,, aw)ll du,
M, (s, X. az: 10
and Q(T. a; D) = h(xx a). then Q Q on [0, T|R XR² Furthermore, the martingale property of M E (10 Ti)is equivalent the following orthogonality condition:
E" [" & [aQ(t, [dQ(t, X,, ax; ) + X,, ar) dt A1 X, at) 1 = 0,
(10)
2
for any test process & E (10, TI: Q(: x.,a.; V)).
Proof can be found Appendix B.I summary, the martingality of the process defined in Equation & under given score V is both necessary and sufficient for Q to be the corresponding action value function.
4.2 Score Evaluation of the Q-Function
We now discuss how the HJB equation can be used to design Q-learning algorithm for estimating Qxx, a; V using sample trajectories. A number of algorithms can be developed based on two types of objectives: to minimize the martingale loss function or to satisfy the martingale orthogonality conditions. Following (23), we leverage the martingale orthogonality condition, which states that for any 7 > 0 and a suitable test process E.
E J'' (1000, X,, V)++ (r(,, X1,
- 11ll X,, az)
de - =0. 0. )
(11)
To approximate the Q-function, we consider a parameterized family Q" V) where E C Le (in principle, we need least Le equations martingale orthogonality conditions in order to fully
5

---

02" determine 0.) and choose the special test function & 00 (1, Xe, de; v) Stochastic approximation 38 leads to the online update:
aQu 0 - + cxp 00 (1, Xx, az; v) (dQ(1, an; + [r(1, Xr, Xr, - AlI X., at) a1) (12)
where Og a learning rate. This recovers the mean-squared TD error (MSTDE) method for policy evaluation in the discrete RL [3]. We must, however, stress that testing against this specific function theoretically sufficient to guarantee the martingale condition. Additional discussions are provided in Appendix D
4.3 Policy Optimization via Matching the Score to the Q-function
Now extend the analysis to the infinite-horizon setting. The dynamics of the state and action processes are given by:
dX, = (X,, at) dt + ox (X,, ar) dB," da, = w(X,, az) dt + oa(X,, dB,,
(13)
and the following discounted Q-function under any given Tr:
(xx a; V) R.P V.
e e- r(Xx, 1 2 (X a ds X. x, at
(14)
where 3 > O is discount factor that measures the time-depreciation of the objective value (or the impatience level of the agent) and the optimal Q-function Q a) supwen Q a; VV
Note that in this case, the Q-function does not depend on time explicitly. As a result, there is no terminal condition, but instead have growth condition E'Ie [e TTle--tQ "Q((X, ) + 0 as f co. Again, using the dynamic programming principle, we have
(Q((x,)) (Q(z, + r(z,0) (1 a)) =0 Well
sup
(15)
We now introduce an alternative method for updating policies based on Q-function estimates that avoids the use of policy gradients. The following result serves as a score improvement theorem, analogous to the classic policy improvement theorem in reinforcement learning.
Theorem 2. Let Vo be any given score function and let the associated Q-function Q(::; V E
(RR n x Rdd) n (IR" x IR.). Suppose further that the score function 1 defined by Wpr
V. (x, at (r for some 0 is admissible, Then Q a; 4 2 Q(2, a; W) (z, a) R" Rd
Proof can be found in Appendix B.2
35 similarly constructed score function V but their approach only determines the direction of the policy update, without specifying the magnitude of the update vector. This result implies that iteratively updating the score function by aligning it with the action gradient of the Q-function leads to monotonic improvement in the Q-values. In other words, setting V + X-1V Qx, a) guarantees an improvement in the resulting Q-function globally. Figure [saurs visual description of Theorem 2 and the implied policy update direction via CQSM. Building on this theoretical foundation, we now describe how to implement sample-based update using a parameterized score function " with the parameter v E L., To match the direction of V a (zx a), we define the update as: U E arg min } "2 X V.Q a) This yields the CQSM as shown in Algorithm
Policy sampling. Consider the dynamics of 13 for a fixed state and setting o (2, a) = v21a- Given certain conditions, under appropriate regularity conditions, the stationary distribution of the action a, as action distribution corresponds to a Boltzmann distribution over actions: <aaa)) = 2 Q( where Z = -zz e Q(+aa) da. This gives rise to soft optimal policy, where the action distribution is shaped by the Q-values, rather than relying on the soft Hamiltonian or auxiliary q-functions used in the soft actor-critic literature (e.g., [24 25 ). However, for general oa (z, a), the stationary distribution ((aar) may not have a closed-form expression Appendix E.
for any fixed E R denoted π (a]zr): T(alx:) Q(r,)) That is, the stationary
More details about action sampling can be found in

---

Optimal Q
Optimal
v_Q
V v.Q
d,
Figure 1: The left image shows randomly initialized score function vo and the right shows the updated score y after one step. If discrepancy exists between the score v (green vector) and the action gradient VaQ(x, a) (blue vector), then aligning Vi with V.Q yields a strict improvement in Q-value (x, a).
Here highlight several hyperparameters: the trajectory truncation parameter (time horizon) T (needs to be sufficiently large); the total sample size N or the sampling interval A, with N A = T. We define the observation times as tx k At, O..... the simulated environment, denoted as Environmentar.
which data collected from
Algorithm 1 Continuous Q-Score Matching (CQSM)
1: Inputs: initial state zo, time step At, initial learning rates oee, Oty and learning rate schedule function z(-) (a function of time), the action value function (-,-,, the score function ( the test function ((I-)e a.Axe). the initial parameter fy vo. and regularization parameter A.
2: Required: Enviromment simulator (I', r) = Savirommentxx((2
3: for k 0. N do
4: Sample action via iterative denoising: a² ~ (0, 7) + a., using score V.: a ~ (xx)
5: Simulate environment step: (a', r) Environment (Ir, a). Update state: Pias1 r
6: Sample new action a' ~ (Tx') via denoising with TT. Store Atat1 + a'
7: Compute test function: Er, E(Ittgsk (tg:k)
8:
Compute temporal difference:
A = Q" (x, + +r(x 2 (1'' -/////033000000
9:
Compute parameter updates:
A Eu 8, AU= (1 - -v((a,
O5"
Ou (z,
Update parameters:
- 0 I(kAt)aa =-u+((AA)a,..
11: Set <- x'
12: end for
5 Experiments
In this section, present numerical evaluations LQ control tasks. Additional experimental results are provided in Appendix(F We compare the performance of our proposed CQSM algorithm against continuous time policy gradient (24) and continuous time little q-learning methods (25). Below, briefly review these baseline methods.
CT-RI policy gradient: Given an admissible policy, this method first performs policy evaluation to estimate the corresponding value function. It then computes the policy gradient
7

---

as: g(t, z; () 0 [t, z; TT) (J is a value function). 24 transforms policy gradient into de policy evaluation to develop policy gradient algorithm.
CT-RI q-learning: The little q-function is defined as the first-order derivative of the Q-function with respect to A. Policy improvement is achieved via (a)t, exp (t,,,a) ] exp (t,,a)) da leading continuous-time q-learning theory.
Linear- Cndear-uuadratcc Stochastic Control. We now focus on the family of stochastic control problems with linear state dynamics
bx (1, a) Ax + Ba and ox ox(x, Cx + Da, = V2,
(16)
where A, B, C, R and the quadratic reward
r(a,, a) - M x² + Rea + N 2
Px Px+ P
(17)
2
where M 2,N 0. R, P,P E If O. then one smooth solution to the HJB equation
1 0a
8Q(x,a) Qub(I,, a)
2 x 2
-00 r( = 0,
(18)
2x
given by Q(x, a) 3koo² + kI + hhga + kza + kyra kg where
ko = x(3-2d 103-21-03
C2j ki 3-2A-03
ki MB K3k44
p
3-1
l 2
3 Xv
32
+ 3 IN - -2
BP+P'(3AA)
(19)
K3 (3 ()
X(O ka- )
k1 -08-24-C BB AP 2A B2 DDM 24))))))
hs 2 KG
2λ8
For the particular solution, we can verify that hy < O. To ensure Q is concave, a property essential for verifying that this function indeed corresponds to the action-value functior we impose the additional conditions Ko < O, kok2 K3 > O. Next, we state one of the main results of this paper.
Theorem 3. Suppose the dynamics and the reward function are given by (T6) and (77), respectively.
Then, the Q-function is given by Q a) = kg2x ky + 2 krya? kzaa + k4ra + ka where ku, K1, 12, kis R4, ki are as in [19. Furthermore, the optimal score function takes the form:
V V' x Ak2aa + ha kar)).
(20)
Additional details Theorem are provided in the Appendix
C
In our simulations, to ensure the stationarity of the controlled state process, we use the following model parameters: A 1,
1, M N p' 2, 1. 1, 0.1. We parameterize the Q-function as Qo 1 0 2 + 00 +0 and the corresponding score function V (xr a) = - a + 4/2 + v3. Using these parameterizations and the model setup, the optimal parameter values are computed as:
0" =[--059047134. - - 0.23069812, -10.::81......4411161611111 -0.46111679 -.335224157, -0. 15119900 0.17312350] v" = [1.529133555 1.51119000. -3.5624157
Implementation Details. The learning rate initialized ag a. 0.01 and decay according to [(t) max{1.v|oge}} To evaluate performance and stability, each experiment is repeated five times with different random seeds for sample generation. The parameter vector 0 is initialized as zero and is initialized in the range [0, 1]. The corresponding optimal values 0* and v² are then used as baselines for comparison. The time cost for each experiment is 352 seconds on a computer with Intel Core i5- 10500 CPU and 32G Memory.
the HJB equation has additional quadratic solution, which, however, is convex.

---

Performance Results. Each experiment is repeated ten times with different random seeds. Figure 2 Iillustrates the convergence behavior of the proposed CQSM algorithm for one realized trajectory with time step A/ 0.1. Both the Q-function and score function parameters gradually approach their theoretical optima except 22 03, 2/- Note that these parameters are closely tied V.Q. Our method requires the estimation of both the Q-function Q and its gradient V Q. This dual estimation introduces additional variance and bias, potentially leading to inaccurate policy updates.
Furthermore, we compare the performance of our CQSM algorithm against two benchmark methods in terms of the running average reward obtained during the learning process. The other two algorithms are the PG-based algorithm proposed in ( ([24). Algorithm 3) and the little q-learning algorithm presented ([25), Algorithm 4). Figure 3 presents the running average rewards and standard deviations for all three methods under three step sizes, At = 0.01, 0.1, Our proposed CQSM consistently outperforms the baselines in the early stages of training, achieving higher rewards more quickly than both PG and little q- g-learning. After a sufficient amount of time, all methods eventually stabilize similar average reward levels.
0,2
01
Pat
0.8
Tie Vaderry
-0.1
28003
40989
.0090
80090
19900
21006
20000
J0068
S0000
190030
20000
40890
2900
100999
Time
Tim
(a) The path of learned 0o. 01, 02 (b) The path of learned 0s, 04, 0s (c) The path of learned Up, 01, U2
Figure 2: Paths learned parameters the CQSM reinforcement learning algorithm described in Algorithm I A single state trajectory of length T 10 is generated. The dashed lines indicate the optimal parameter values. The shaded regions represent the standard deviation the learned parameters across these runs, with the width of each shaded area equal to twice the corresponding standard deviation.
-2.
Time
Tla
(a) At= 0.01
(b) At = 0.1
(c)
At=1
Figure 3: Running average rewards of three RL algorithms. A single state trajectory of length
T 10a is generated and discretized using three different step sizes: At 0.01 in panel (a), At = 0.1 in panel (b), At = in panel (c). For each setting, we apply three online algorithms: Policy Gradient described in Algorithm 3 in 24. q-Learning described in Algorithm 4 in 25 and CQSM described in Algorithm[] [[] We plot the mean running average reward over time and the shaded areas represent the standard deviation across the runs.
Conclusion
In this paper, we introduce Q-function framework for continuous-time stochastic optimal control problems with diffusion policies. By using the dynamic programming principle, we derive the associated HJB equation for the Q-function. Building on this and utilizing the martingale orthogonality condition, develop the CQSM algorithm. We further demonstrate the effectiveness of COSM

---

in an LQ setting, showing promising results compared to existing continuous-ting time reinforcement
learning algorithms.
Several interesting directions remain for future work. For the finite-horizon case, extending the approach to handle important portfolio selection with mean-variance objective poses challenging problem due to inherent time inconsisteney. Another promising direction is the optimization of the diffusion term a. which could lead improved exploration and performance, Furthermore, a theoretical convergence rate analysis of CQSM could offer deeper insights and guide further enhancements the algorithm.
Acknowledgements
We gratefully acknowledge financial support from the Key Project of National Natural Sci- ence Foundation of China 72432005, Guangdong Basic and Applied Basic Research Foundation 2023A1515030197.
References
[1] Alain Bensoussan, Guiyuan Ma, Chi Chung Siu, and Sheung Chi Phillip Yam. Dynamic mean-variance problem with frictions. Finance and Stochastics, 26(2):267-300, 2022.
[2] Alvaro Cartea, Sehastian Jaimungal, and Jason Ricci. Algorithmic trading, stochastic control, and mutually exciting processes. SIAM review, 60(3)667-7733 2018.
[3] Shuping Chen, Xunjing Li, and Xunyu Zhou. Stochastic linear quadratic regulators with indefinite control weight costs. SIAM Journal on Control and Optimization, 3665):1685-1702, 1998.
[4] Min Dai, Yuchao Dong, and Yanwei Jia. Learning equilibrium mean-varianc varrivaiance strategy. Mathe- matical Finance, 33(4): 101-166611110 1212, 2023.
[5] Shutong Ding, Ke Hu, Zhenhao Zhang, Kan Ren, Weinan Zhang, Jingyi Yu, Jingya Wang, and Ye Shi. Diffusion-based reinforcement learning via q-weighted variational policy optimization. arXiv preprint arXiv: /6/73, 2024.
[6] Carles Domingo -Enrich, Michal Drozdzal, Brian Karrer, and Ricky TQ Chen. Adjoint matching: Fine-tuning flow and diffusion generative models with memoryless stochastic optimal control. arXiv preprint 24Xi::2409.0866. 2024.
[7] Kenji Doya. Reinforcement learning in continuous time and space. Neural computation, 12(1):219-245, 2000.
[8] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International conference on machine learning, pages 1329-1338. PMLR, 2016.
[9] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.
[10] Xuefeng Gao, Zuoquan Xu, and Xunyu Zhou. State-dependent temperature control for langevin diffusions. SIAM Journal on Control and Optimization, 60(3):1250-1266 1268, 2022.
[11] Xuefeng Gao, Jiale Zha, and Xunyu Zhou. Reward-directed score-based diffusion models via q-leamnng. arXiv preprint arXiv:2409.048.. 2024.
[12] P. J.Jamson Graber. Linear quadratic mean field type control and mean field games with common noise, with application to production an exhaustible resource. Applied Mathematics & Optimization, 74:459-486, 2016.
[13] Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep q-learning with model-based acceleration. In International conference on machine learning, pages 2829- 2838. PMLR, 2016.
10

---

[14] Xin Guo, Anran Hu, and Yufei Zhang. Reinforcement learning for convr- models with jumps via stability analysis of feedback controls. SIAM Journal on Control and Optimization, 61(2):555-777, 2023.
[15] Xin Guo, Renyuan Xu, and Thaleia Zariphopoulou. Entropy regularization for mean field games with learning. Mathematics of Operations research, 47(4):323--3260, 2022.
[16] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. Pmlr, 2018
[17] Yinbin Han, Meisam Razaviyayn, and Renyuan Xu. Stochastic control for fine-tuning diffusion models: Optimality, regularity, and convergence. In Forty-second International Conference on Machine Learning, 2024.
[18] Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6840-6851. Curran Associates, Inc., 2020.
[19] Yilic Huang, Yanwei Jia, and Xunyu Zhou. Achieving mean-variance efficiency by continuous- time reinforcement learning. In Proceedings of the Third ACM International Conference on Al in Finance, pages 377-385, 2022.
[20] David Ireland and Giovanni Montana. Revalued: Regularised ensemble value-decomposition for factorisable markov decision processes. In The Twelfth International Conference on Learning Representations, 2024.
[21] Haque Ishfaq. Guangyuan Wang, Sami Nur Islam, and Doina Precup. Langevin soft actor-critic: Efficient exploration through uncertainty-drived critic learning. arXiv preprint arXiv:2501.17827, 2025.
[22] Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.099: 09991, 2022.
[23] Yanwei Jia and Xunyu Zhou. Policy evaluation and temporal-difference learning in continuous time and space: A martingale approach. Journal of Machine Learning Research, 23(154):1-55, 2022.
[24] Yanwei Jia and Xunyu Zhou. Policy gradient and actor-critic learning in continuous time and space: Theory and algorithms. Journal of Machine Learning Research, 23(275): 2022.
[25] Yanwei Jia and Xunyu Zhou q-learning in continuous time. Journal of Machine Learning Research, 24(161):1-61, 2023.
[26] Jeongho Kim, Jaeuk Shin, and Insoon Yang. Hamilton-jacobi deep q-learning for deterministic continuous-timg time systems with lipschitz continuous controls. Journal of Machine Learning Research, 22 2021.
[27] Pankaj Kumar. Deep reinforcement learning for high-frequency market making. In Asian Conference on Machine Learning, pages 531-546. PMLR, 2023.
[28] James-Michael Leahy, Bekzhan Kerimkulov, David Siska, and Lukasz Szpruch. Convergence of policy gradient for entropy regularized mdps with neural network approximation in the mean-field regime. In International Conference on Machine Learning, pages 12222-12252. PMIR, 2022.
[29] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: 0509.0277, 2015.
[30] Haitong Ma, Tianyi Chen, Kai Wang, Na Li, and Bo Dai. Efficient online reinforcement learning for diffusion policy. In Forty-second International Conference on Machine Learning, 2025.
11

---

[31] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforee- ment learning. In International conference on machine learning, pages 1937. PmL.,, 2016.
[32] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 1rXiv:1312.5602, 2013.
[33] Rémi Munos and Paul Bourgine. Reinforcement learning for continuous stochastic control problems. Advances in neural information processing systems, 10, 1997.
[34] Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgesou, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, et al. Imitating human behaviour with diffusion models. arXiv preprint ar arXiv:2301.10677, 2023.
[35] Michael Psenka, Alejandro Escontrela, Pieter Abbeel, and Yi Ma. Learning a diffusion model policy from rewards via q-score matching. In International Conference on Machine Learning, pages 41146-41 182. PMLR, 2024.
[36] Martin I. Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
[37] Moritz Reuss, Maximilian Li, Xiaogang Jia, and Rudolf Lioutikov. Goal-conditioned imitation learning using basee diffusion policies. arXiv preprint arXiv:2304.02532, 2023.
[38] Herbert Robbins and Sutton Monro. A stochastic approximation method. mathematical statistics, pages 400-407, 1951.
The annals of
[39] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1771v:1707.06347, 2017.
[40] Tim Seyde, Peter Werner, Wilko Schwarting, Igor Gilitschenski, Martin Riedmiller, Daniela Rus, and Markus Wulfmeier. Solving continuous control via q-learning, In The Eleventh International Conference on Learning Reprexentations, 2023.
[41] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning. pages 387-395. Pmir, 2014.
[42] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[43] Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 44, 1988.
[44] Richard S Sutton, Andrew G Barto, al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
[45] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems, volume 12. MIT Press, 1999.
[46] Corentin Tallec, Léonard Blier, and Yann Ollivier. Making deep q-learning methods robust to time discretization. In International Conference on Machine Learning, pages 6096-6104. PMLR, 2019.
[47] Akshaj Tammewar, Nikita Chaudhari. Bunny Saini, Divya Venkatesh, Ganpathiraju Dharahas. Deepali Vora, Shruti Patil, Ketan Kotecha, and Sultan Alfarhood. Improving the performance of autonomous driving through deep reinforcement learning. Sustainability, 15(18):13799, 2023.
[48] Wenpin Tang, Yuming Paul Zhang, and Xunyu Zhou. Exploratory hjb equations and their convergence. SIAM Journal on Control and Optimization, 6006):319-33116, 2022.
12

---

[49] Arash Tavakoli, Mehdi Fatemi, and Petar Kormushev. Learning to represent action values as hypergraph on the action vertices. In International Conference on Learning Representations. 2021.
[50] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033 IEEE, 2012.
[51] Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_control: Software and tasks for continuous control. Software Impacts, 6: 100022 2020.
[52] Haoran Wang, Thaleia Zariphopoulou, and Xunyu Zhou. Reinforcement learning in continuous time and space: A stochastic control approach. Journal of Machine Learning Research, 21(198):1-34, 2020.
[53] Haoran Wang and Xunyu Zhou. Continuous-time vaan-variance portfoioo selection: rein- forcement learning framework. Mathematical Finance, 30(4): 2020.
[54] Yinuo Wang, Likun Wang. Yuxuan Jiang, Wenjun Zou, Tong Liu, Xujie Song, Wenxuan Wang, Liming Xiao, Jiang Wu, Jingliang Duan, et al. Diffusion actor-critic with entropy regulator. Advances in Neural Information Processing Systems, 57:54183-54200 2024.
[55] Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023.
[56] Christopher John Coriish Hellaby Watkins. Learning from delayed rewards. Ph. D. thesis, Cambridge University, 1989.
[57] Jiongmin Yong and Xunyu Zhou. Stochastic controls: Hamiltonian systems and HJB equations, volume 43. Springer Science & Business Media, 1999.
[58] Hanyang Zhao, Haoxian Chen, Ji Zhang, David Yao, and Wenpin Tang. Score as action: Fine- tuning diffusion generative models by continuous-time reinforcement learning. In ICLR 2025 Workshop on Deep Generative Model in Machine Learning: Theory, Principle and Efficacy, 2025.
[59] Hanyang Zhao, Wenpin Tang, and David Yao. Policy optimization for continuous reinforcement learning. Advances in Neural Information Processing Systems, 36: I 3637-13663, 2023.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
is contained in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: is showed in section 4.2] This test function theoretically not sufficient to guarantee the martingale condition. Be careful when choosing test function, or choose other methods.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-specfiiaatinn. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [Yes]
Justification: Assumptions are in the section 3 5 and Proofs are provided in the Supple- mentary Materials.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
All the needed information is in the section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
15

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: For data, please see the section The code will be released upon paper acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ httpppscccppullccguuudddeo Coaee//ddeeeSbmmmmma ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental details are shown in the section Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Experiments were repeated 5 times, and the mean and variance of parameters and reward were plotted in section 5
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Compute resources are shown in the section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We follow the Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
This work a foundational research.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Both baseline codes for comparison are publicly available under the open- source licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt
package should be provided. For popular datasets, paperswi thcode. cww/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The details of the code will be released upon paper acceptance.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
20

---

A Diffusion RL: Problem Formulation and Well-Posedness
Listed below the standard assumptions to ensure the well-posedness of the stochastic control problem in (T
Assumption 1. The following conditions for the dynamics and reward functions hold true: (1) bx, xx, oa, h, Vor, Voh all continuous functions in their respective arguments;
(2) bx, ax, Ga are globally Lipschitz continuous in (z, a), i.e. for yp {b,ox), there exists constant C > 0 such that
,,, a' E B.: (3) bx, ox are linear growth continuous in (Ir, a), ie., for E (b,ax), there exists constant
[1, x; a) - -l,, , d') S C((r-= - + ||a d' (2), VI [0, E ER
C > 0 such that
1| a)ll S C(1 + (all)). VE E [0, T) EERE
(4) a. is bounded for any (x, a). i.e, there exist constants C > such that llaa(t, x a)lla S SC [0. [], x E R" a E Idd.
(5) r, h. V Var. Vah have polynomial growth in (I, i.e, for E fr,h, Var, Vah} there exist
constants C > O such that
lo(t, x, a)l S C(1+ + +|ll)) Vt RR,,
Lemma 1. Suppose {a, 1 0} follows
da = by dl + or dB; 0,
with C. esssupr, |arl oc. If So 1,,12 ds < xo for all I 2 0,a a E Rd, then there exists
constant C > 0, which is independent of and ap, such that
EP sup OSAT
2 c 100 VT VT 20.
(21)
Proof. By the elementary inequality
(a+b+ad < (imax(a,b,c)) < 3² +32 +
have
|ool + sup / 1b./ds + sup 0SIST
''' o, dB3
E
sup LOSITT
larl
=F²
0515T
/ dB
<E²²
|0ol +
[by d8 + sup OSIST
0
/ o, dB
$32 [0ol
1 + 33
16, + +
sup OSiST
1EP [ |o [o212da ds
<32 c3210012 + 32
16.12 ds + +3C²
100 + 32 1 ds + CC
<C (1+ (aol))
where the second to last inequality due the Burkholder-Davis-Ooe inequality. This proves 21
Lemma 2. Let Assumption hold, the solution of state SDE 7 satisfies the condition
EF
sup
< (1 1xol?),
(22)
LOSEST
for some constant C 0.
21

---

Proof. Based on the proved growth condition on ox, Cauchy-Schwarz Schwarz inequality, and Burkholder-Davis Davis-Gundy inequalities, we obtain
E
sup LOSIST
xx|
I bx (x, as) 2
L' ox(88, xx, d.) OstST
[00 2 + sup USAST
+ sup
("(
SC
|2ol 2 + C2
sup
1 la512 laao1 ds
00
$Cs(1 + (mol?) + C2
Jo
sup E. | ds. 05+5#
Applying Gronwall's inequality, obtain the desired result.
Theorem 4. Let Assumption 7 hold, then there exists constant C1 >> 0 such that the Q-function
satisfies
Q(1,2, a)) S Ci + ||0|| + ||a|22)
for all t E [0,T], RR, E Rd Finally, the Q-function is finite.
Proof. Let v = Chere C' is a constant. It then follows from Lemma 1 aad Lemma 2 that
a(),
((; - A 1 as 2l) ds + h(XT, (r) X, x. ae
>
2 E /." (-C(++)xX |x,ll (|as122) 00 ds X, 5,
C²(1 + ||0||| | ||a|12)
for some constant C' independent x, a. On the other hand, for any V IT, we have
r( r(Xo, dg) A 11(X a ds + 1(Tx ar) Xx x, a, a
Q(, =E²
SE. J. (C(1 + |x + [la. X, = x, 0,
<C" (1+ + ||ا|| + ||a||2)
for some constant C independent of The final result is evident. The proof is complete.
We have indeed established in the above that
L.
E
as) ds < c0.
(23)
B Proofs of Martingale Characterization and Score Improvement Theorem
B.1 Proof of Theorem
To show M. Q(s, 1i Se (u,, an) 1/1l Xu, [.) du is martingale, ob- serve that
J. r(u, Xur aw) / A1l Xu, au) ds h(Xr, (()..oT) or)
M, E
1."' Xu) 1 (u, -
(24)
X a || 2 du
= [MT]FF)
22

---

where we have used the Markov property of the process a,), $ ST. This establishes that M is martingale.
Conversely, if M is a martingale, then M, = E² [M which is equivalent to
r(u, r(u, Xu. Xu: an) _ (u Xux an am)l du + Q(T, XT, (r)
Q(s uz: asi V) =BP
=E J.a (rlu, Na, (u an)
(25)
du M(X)) Fs 2 (u, X an)
(s, X., uo; V), 8 E [1,T).
Letting& f, we conclude Q x. a; V Q(t, x. 0z VD) The "only if' part evident. To prove the "if" part, assume that dM, A, dt + C. In particular,
in our case, A,- = LQ(I, x, V) + r(,, Xl, a.) 1 \L, az) and C, =
(8Q G(I, Z.).
A, E (10 T follows by assumption (Q C².2,2((O, T) x R x R n C((O, x R" x R))
and Theorem For any I s < S T, take & = sgn (() if E [s, s and & otherwise. Then
&. d.M = BB/ L (IA,) dt + &.C, dB,) = EF
E 0 0= 0-E²
|Ai dt,
(26)
where the expectation of the second term vanishes because [EC] S |C'| E (10, and hence EP So &,C, dB, is a martingale. This yields A, = almost surely, and thus M is a martingale.
B.2 Proof of Theorem 2
Fix (x,0) E R x Rd, applying Ito's formula, we have
e <=x0((x,; = =) + ( - (- V.Q ax)
+V.QT v((x; + (2x + デリ (o.0 v:Q) } dr
+ +f. c - Q ax (X ax) dB, + 8r V.QT oa r, a.) dB,
(27)
Define the stopping times Tn = inf(s t: || 2 n, |a..l2 2 n), for n 2 1. Then we have
B(sATw)
MNTT, v))
Xx Xx
a, = a st ,Q(
E f.are e ( -3; v) + V.QUT b V ((X,,ar)
+ | (oxax + (oo0 dr Xx ==
(28)
On the other hand, by standard arguments and the assumption that Q(, v) is smooth, we have
BQ(z, a; v (v=QT (x(xx, + V (z,
1 + tr (xxx\(2)) + 2 (0.07 V2Q) (2 + (()
1 2 1 = 0.
(29)
for any (2, E RR" x Rd.
follows that
BQ(I,a;)) v) sup
sup { V.Q VT -x(xx + V.QT (2,
(30)
1 - (o. (2Q) + ,(( a) - = 0..
(oxo +
23

---

Notice that the minimizer of the Hamiltonian in 300 is given by vp = Av v) for some 0. It then follows that Equation (28) implies
R
((eATa) Q(xaNT,, ASAT,; v) X, 2i 50 aa
(31)
panin -((--() (r) > 2114113 dr X, 0,
20 31 Q(11 az v) E.P
Sending π + 20, we deduce that
E. e { ((x: oz: 1) = 2, =a
J. ((--1) r(zT,, ar) 1 dr X1 x,
(32)
Ze B1 (z, v E
Noting Lemma
Lemma and Q is polynomial growth, we have
lim Eº |e Q(x as: V) X₁ = 2, as = al
8-400
(33)
< sup E. (e 8a Q(x., asi v) I, a =0
$-200
and applying the dominated convergence theorem yield
(z a; V) S EP
[.. e-rr
(
X 1 dr X1 0,
Q(I,a; (34)
C Derivation of Linear-Quadratic Stochastic Control
The following derivation corresponds to Section 5 of the main text.
Assumption 2. The discounted rate satisfies 3 > 2A + C².
This assumption ensures a sufficiently large discount rate, which guarantees that lim inf e 37 - [Q(X7, dy; 4)] 0 for any score v, thereby ensuring the correspond- ing expected reward remains finite.
By HJB equation
1 Q:
1
1
8Q(x, - Qub(I,a)
x00
0000
r(x, 0,
(35)
2
2
2
have
B
1
1
x²
kin
Alio
-
C2 + M/22
2
2X
2
highis
x:
8kg
Ak
+ P =0
X
1
a²
1/
K
B
h
12
+ N/2 = 0,
2
2X
2
k2ha
a:
3kg
Bky
p' = 0,
X
va
Bh4
B.B
K4A
k2kks
KD R=
1
Cons Bks 12
22
0.
2)
By 2 term:
M
Ko X(3 2A
12
(36)
(2)
3 2A C
and substitute ko to a² term, we obtain
B
1
1
M
k2
k1B
D
12
+ NN/2 0,
(37)
2
21
2
ABB
2A C2)
3 2- C²
3
M
h2
13 + N/2 = kuB +
D
12
=
(38)
2
2X
2
X(3 2A C2)
B 2A C
24

---

First, we consider D * O. Hence, we have
3
g2
1/2
2 x- \v (N )
(39)
h4 1(B 2A C²) + NB 2A C2
D2(DDAM 2X(3 2A C2
(2) )
D
(40)
By term, determine the value of and satisfies the following bounds
g 1
X (N 2 2 0,
(41)
A² 2A C2 DDMM + 2X(3 2A (2) 2
(42)
If D 0, then
3X
82
2
N
2:2
X1
his B
(43)
and l4 satisfies the following equation:
B
3
82
2
N
kaB)
MB
A +
K4 +
+
R = 0.
(44)
X(3 2A)
X
2
B 2A
Furthermore, by x term and a term, we have
1
P
k1
kiak4
(45)
1(3 A)
3 A'
BP + P' A)
03
(46)
(3 A)
3 x(3- 13
K4 1 1/2
Finally, we have
2Ak2 + k-
k; 2A,
(47)
Therefore, the optimal score function takes the form:
T (z" a) - - (kza + hs + k(r).
(48)
D Continuous Actor-Critic Q-Learning Algorithms
Score Evaluation of the Q-Function. We provide complete description of how the HJB equation used to construct the continuous-tin time Q-learning algorithm. For estimating Q(z, a; V a number of algorithms can be developed based two types of objectives: to minimize the martingale loss function or to satisfy the martingale orthogonality conditions. We summarize these methods in the Qelerrning context below.
(1) Minimize the martingale loss function:
E. J''
((XTTar) (2 at) + f"' (M(sx - Xs, Xx, ds di
]
2
2
(49)
This method intrinsically offline because the loss function involves the whole horizon [0, TJ. We can apply stochastic gradient decent to update
0 レ 0 + as ('
aQ
(t, Xx, az G.)
00 T
(50)
OT
v - + 0g
\V ax)
Ov (s, a,) dsGer dt,
25

---

where GGT (xXr,ar)-QQ( Xz, a+) + Se (r(x, x.,
| (s (x, ax)l)) ds. We present Algorithm| 2 based on this updating rule. Note that this algorithm is analogous to the classical gradient Monte Carlo method or TD(1) for MDPs [4] because full sample trajectories are used to compute gradients.
(2) We leverage the martingale orthogonality condition, which states that for any T >> 0 and a suitable test process 5.
E² Ee (1, + + ((t, -
& AA (t, X1, at) xx,
dt O.
(51)
We use stochastic approximation to update 0 either offline by
& (x 4) ((t,
1 ul Xt, (tx
0 + 0 as
dt
(52)
2
or online by
1 rtt, X2, az) - 11 2 (t, X1, at)) at) ary.
0 - 0 + +
v) +
(53)
Typical choices of test functions are aq Q-learning algorithms based on stochastic approximation. However, relying solely on this specific choice does not, in general, guarantee satisfaction of the full martingale condition. Moreover, the convergence of the resulting stochastic approximation algorithm is not assured without additional assumptions. As discussed in [23], the selection of test functions must be carefully tailored to the structure of the Q-function, highlighting the need for more robust and theoretically grounded choices in continuous time settings.
00 & So ppp--a00 0Q du ds, 0 < p which lead to
(3) Choose the same type of test functions & as above but now minimize the GMM objective functions:
EP 1 & (1," r(1, x ao )al 100-g((( (1, xi" o0% )di BJ (1, X; )dl
A&E? [ Eo (a XT, of jtt 100--gtt,, Xt, ar
== a jdt 3J (t, )at)
(54)
where Ag E SLe, Typical choices of these matrices are Ag = Ir,, or Ag = (R EET d()) Again, we refer the reader to [23] for discussions on these choices and the connection with the classical GTD algorithms and GMM method.
Score Gradient.
We aim to compute the score gradient u)) = 8 (xx as V²) E R at the Do current actaon pair aI. Based on the HJB of the Q-function, we take the derivative on both sides to get
O5" (x, a) =
Cg(x, a; Bg(a, v) (V.Q(r, (x,
(55)
Or
Thus, a Feynman-Kat formula represents g as
g(x, az v) = 1P [. - (V.V((X V²) XV an))
OV" (X ag) X, = x, a:
Ov
(56)
To treat the online case, assume that v" is the optimal point of Q(x, a; (r") for any (æ, a) and that the first-order condition holds (c.g., when v² is interior point). Then g(z, v*) = 0. It follows that
0 == E. U. Tee TeVVQ((X asi V() X a.))
O0 (X, a. ds X, =2
(57)
dr
for any 7) E (1, Q(: X., a.; V)). If we take 7), = - Bs with g(x, a; 8 e)). More importantly, besides the flexibility of choosing different sets of test functions,
then the right hand side 57 coincides
26

---

provides way to derive system of equations based on only past observations and, hence, enables online learning. For example, by taking 7,s on [T, oo), (57) involves sample trajectories up to time T. Thus, learning the optimal policy either offline or online boils down to solving system of equations (with suitably chosen test functions) via stochastic approximation to find v". Online learning of (57) is the same as the update rule: u E arg min / (x, a) x - (zQ a)l??
57
Here we present Offline Continuous Q-Score Matching (CQSM) Martingale Loss Algorithm in the infinite-horizon setting.
Algorithm 2 Offline Continuous Q-Score Matching (CQSM) Martingale Loss Algorithm
I: Inputs: Initial state To, horizon T, time step At, number of episodes N, number of mesh
grids K, initial learning rates agg, ay and learning rate schedule function I(-), functional forms
of parameterized action value function (:5-) and score function Tp" (:.)) and regularization parameter A.
2: Initialize 0,
3: for episode j 1 to N do
4: Initialize k = O. Observe initial state xo and store It, Io.
5: Choose action by iteratively demoising a" (0, + d using Vr" arz I² (xtx);
6: Step environment (rt, Ttata = env(ata)
7: Obtain observation {at, Tru, JK.,,.,KKK1
8: For every l = to K 1, compute
K-1
G = - -e- (I and +
Σ 1m
e
Ml |r(zt,) an) [(( 2 11 (1)
(58)
2
9: Update and by
K
oQ
0 - 0 -+00
(Ete ase TTt.
(59)
00
k=0
K
K-1
NV'
- u 0+1(j)o
Σ
Σ
Avy
Axt 0t;)
Ov (1t, )e GUTAL.
(60)
k=
t=
10: end for
E
Scored Based Diffusion Models
Consider the forward SDE with state space Y, E Rd is defined as
dy, = [t, Y,) dt + g(() dB, , Yo Tasta
(61)
where f R x Rd + Rd and R + R4. Denote (tt.-) -) as the probability density of Y,.
Set time horizon T > 0 to be fixed, and run the SDE 61 until time T to get Y1 (T, .). The time
reversal yrrev = - for 0 < 1 & satisfies an SDE, under some mild conditions on and g:
dy = =(-1(T -, tron + 1 (T t)V, v log -t
tt,
tt traa)) dt + ng(g dB"
2
(62)
where V, log (t. y) is known as the stein score function and E [0, 1] is constant. In addition, special but important case by taking = in the (62). this results to flow ODE (42
dy (- - Y, 2 g t) log 7(T -t, yrev) dt,
(63)
which can enable faster sampling and likelihood computation thanks the deterministic generation.
Since the score function V, log ((. y) is unknown, diffusion models learn a function approximation so (t, y). parameterized by (usually neural network), to the true score by minimizing the MSE or
27

---

the Fisher divergence (between the learned distribution and true distribution), evaluated by samples generated through the forward process 61
= arg max Uni(0,)) B (( [t) yv) Vy. ((, vel
(64)
where : [o, 2] + R>o is a chosen positive weighting function. When choosing the weighting functions as A(t) (t), this score matching objective is equivalent to maximizing evidence lower bound (ELBO) of the log-likelihood.
For action sampling, we view the action process as the reverse process. First we set 0, (t, Xt,'') of / to untrained time dependent constants. Experimentally, we set 3 = of and ar = Br, 0r I at. When applied to score matching with denoising diffusion probabilistic modeling (DDPM) [8]. samples can be generated by starting from a" ~N(0, 1) and following the reverse Markov chain below
1 at a'
21) + 0, 2
+
(65)
at
F Additional Experiments
F1
Deterministic Case of LQ Control Tasks
We set C = D = 0 to eliminate stochasticity. Figure F.1 shows the running average reward of CQSM, PG and little q-learning. We observe that the resulting performance is comparable to the stochastic case.
Awurage Reward
0.25
Awargge Reward
0.00
0.00
4.25
-0.25
-0.50
-8.50
-8.75
-1.00
Pebry
-1.00-
oppn Pebey
-1.25
28000
40000
60080
100000
20000
40000
60080
100000
Time
Time
(a) At = 0.1
At= = (b)
Figure F.1: Running average rewards of three RL algorithms. A single state trajectory of length
T 10 is generated and discretized using two different step sizes: At = 0.1 in panel (a), At = 1 in panel (b).
F22 Continuous Control Benchmark Tasks
We evaluate CQSM on continuous control benchmarks from the DeepMind Control Suite. DeepMind Control Suite [51] is set of control tasks implemented in MuJoCo (50). We choose TD3 [9], SAC [6], and Diffusion- QL [55] as three baselines for comparison. Below, we first provide a brief review of prior methods.
Policy gradient methods seek to directly optimize the policy by computing gradients of the expected reward with respect to the policy parameters [45. Deterministic policy gradient algorithms for MDPs
(with discrete time and continuous action space) are developed in [41 (DPG) and later extended to
incorporate deep neural networks in 2 (DDPG). Recent studies have focused on stochastic policies with entropy regularization, also known the softmax method; see for example, 3I (A3C); [39
(PPO).
28

---

TD3: [9] proposed Twin Delayed Deep Deterministic Policy Gradient (TD3), which miti- gates overestimation bias in DDPG by using clipped double Q-learning and delayed policy updates. This yields more stable and accurate policy learning in continuous control tasks.
SAC: The Soft Actor-Critic algorithm
[66] optimizes a stochastic policy that maximizes both expected reward and policy entropy. The policy follows π (a|z) ~ e Q(z,a) and is reparameterized using a neural transformation samples from fixed distribution.
55 integrates diffusion models with Q-learnig by adding term that maximizes action-values to the diffusion model's training loss. The final policy-learning objective linear combination of policy regularization and policy improvement.
Diffusion-QL:
We evaluate CQSM against the baselines (TD3, SAC, and Diffusion-QL) on range of MuJoCo continuous control tasks, from high-dimensional domains (Cheetah Run, Walker Walk, Walker Run, Humanoid Walk) to simpler environments (Cartpole Balance, Cartpole Swingup). Each experiment repeated with 10 random seeds, and we report the average episode return across runs (removing extremely poor results).
As shown in panels (a)-(d) of Figure H22 CQSM matches outperforms the TD3, SAC, and Diffusion- QL baselines, particularly in the early stages of training where it achieves higher rewards. This early performance gain highlights a key distinction between our approach and conventional actor-critic methods. Algorithms like SAC and TD3 may frequently sample actions with low Q-values in the initial stage because they do not utilize the action derivative Qa. while Diffusion-QL. can sometimes get stuck at suboptimal solutions. In contrast, our approach benefits from a more immediate policy adjustment via the learned score function, enabling the policy to better approximate the true optimal policy in the early stages. As result, our method can achieve higher rewards in the early steps. In simpler tasks, shown in panels (f)-(f) of Figure F.2, CQSM achieves performance comparable the baselines, demonstrating both its stability and general applicability across task complexities.
29

---

SAC Diflaazn- C00M
TD2
SAC Offtucon- DL
COSM
800
600
400
200
200
Step
Step
100k
200k
300k
500k
100k
(a) Cheetah Run
(b) Walker Walk
Diffusion CQSM
TDS
SAC
Dittuston QL
COSM
500
100
400
300
200
200
100
Step
Slep
50k
100k
150k
200k
250k
600k
8o0k
(c) Walker Run
(d) Humanoid Walk
TD3
Diflusio-QL CQSM
TOS
SAC Diffusion QL CQSH
800
800
600
600
400
200
Step
Stop
20k
10k
60k
80k
100k
20k
40k
60k
80k
100k
120k
140k
(c) Cartpole Balance
(f) Cartpole Swingup
Figure F.2: Experimental Results Across A Suite of Six Continuous Control Tasks.
30


Reviews:
[{"review_id": "0KzfRUuuSr", "reviewer": "Reviewer_jg1D", "summary": "This paper proposes Continuous Q-Score Matching, a model-free reinforcement learning algorithm that leverages diffusion score functions to facilitate learning Q-functions in continuous-time settings. The approach is validated on a class of linear quadratic control problems, analyzing convergence to the analytic parameter optima as well as comparing performance against a selection of baselines.", "questions": "Please refer to the \"Strengths and Weaknesses\" section, with the main suggestion relating to more extensive experimental validation.", "limitations": "Limitations do not appear to be explicitly addressed.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "- The paper is overall well-written and formatted, the manuscript is quite notation heavy and there is a bit of a disconnect between the presented motivation and the experimental validation\n- Line 20-22: low-level robot control systems often run at 0.1-1.0kHz, which is quite fine-grained. Could you elaborate on which dynamics are poorly captured using discretization?\n- Line 32: there is a line of work on discretizing continuous action spaces to apply Q-learning in  high-dimensional “continuous control” settings (e.g., [1-3])\n- If each experiments only takes 4.5 seconds, why not increase the number of runs from 5 to 10 or more to bring down some of the standard deviation bands?\n- In Figure 2, \\theta_2, \\theta_3, \\nu_2 have not converged to their true value. What would happen when running for 1e5 time steps? How would the baselines perform at convergence?\n- The introduction argues about the necessity for continuous-time RL for real-world applications in autonomous driving, robot manipulation, and high-frequency algorithmic trading. For many of these applications, discrete-time RL has been working well. It would be great to see more extensive experiments that further highlight benefits of CQSM over existing alternatives.\n- Furthermore, the experiments in q-learning [4] evaluated 3 step sizes, for 100 re-runs, and 1e5 trajectory length - can you extend the current experiment to match this setting (also re the above convergence point)?\n- It would be nice to provide a summary of the “Policy Gradient” algorithm used as a baseline in order to make the paper more self-contained\n- The legend entries of Figure 2 are too small, please enlarge them. Also, the “Time” x-labels appear cut off on my end.\n- Minor: line 229, “Then” is capitalized\n\n**References:**\n\n[1] Tavakoli, Arash, et al. \"Learning to represent action values as a hypergraph on the action vertices.\" ICLR, 2021.\n\n[2] Seyde, Tim, et al. \"Solving continuous control via q-learning.\" ICLR, 2023.\n\n[3] Ireland, David, and Giovanni Montana. \"Revalued: Regularised ensemble value-decomposition for factorisable markov decision processes.\" ICLR, 2024.\n\n[4] Yanwei Jia and Xun Yu Zhou. q-learning in continuous time. Journal of Machine Learning 346 Research, 24(161):1–61, 2023.", "was_revised": true, "final_justification": "The proposed updates and additional experimental validation improve the paper, providing further validation on common benchmark tasks. Expanding on the MuJoCo experiments by validating on additional tasks and with additional baselines would further strengthen the paper.", "rebuttal": "We sincerely thank your time and efforts in reviewing our paper and for your valuable comments! We appreciate your positive comment such as \"The paper is overall well-written and formatted\". In the following, we provide detailed responses to the comments point by point.\n\n**Comment (1):** *The manuscript is quite notation heavy and there is a bit of a disconnect between the presented motivation and the experimental validation.*\n\n**Response:** Thank you for the comment! Our presented motivation is:  Q-learning is a cornerstone of discrete-time RL, however, when Q-learning is directly extended to continuous time, the Q-function tends to collapse into an action-independent value function. In Jia et al. (JMLR 2023) [1], they show that the discrete Q-learning algorithm is noisier and slower in convergence speed compared with their proposed continuous PG and little q learning algorithms. Our proposed method can maintain the action characteristics of the Q-function, so our final validation experiment is to compare the continuous Q-learning algorithm with the action-independent value-based RL model. We have simplified the notation, added clarifying remarks, and included a notational summary to improve readability. We also revised the introduction to better align the theoretical contributions with the experimental validation.\n\n**Comment (2):** *Low-level robot control systems often run at 0.1-1.0kHz, which is quite fine-grained. Could you elaborate on which dynamics are poorly captured using discretization?*\n\n**Response:** Thank you for raising this point. Many real-world control tasks require smooth, continuous actions in response to high-dimensional, real-valued sensory input. In applications of RL to continuous problems, the most common method first discretizes time, state, and action and then applies an RL algorithm for a discrete stochastic system. However, this discretization approach has the following drawbacks: When a coarse discretization is used, the control output is not smooth, resulting in poor performance. When a fine discretization is used, the number of states and the number of iteration steps become huge, which necessitates not only large memory storage but also many learning trials [2]. Overall, the resulting discrete-time algorithms are highly sensitive to time discretization [3]. \n\n**Comment (3):** *Line 32: there is a line of work on discretizing continuous action spaces to apply Q-learning in high-dimensional “continuous control” settings (e.g., [4,5,6]).*\n\n**Response:** We thank the reviewer for pointing out these relevant works. Discretizing continuous actions is a common approach to extend Q-learning, but it often struggles with scalability in high-dimensional spaces and relies on discrete-time assumptions. In contrast, our method maintains continuous-time and continuous-action representations. We have added citations for your mentioned works and clarified the distinction in the revised manuscript.\n\n**Comment (4):** *If each experiments only takes 4.5 seconds, why not increase the number of runs from 5 to 10 or more to bring down some of the standard deviation bands? Furthermore, the experiments in q-learning [1] evaluated 3 step sizes, for 100 re-runs, and 1e5 trajectory length - can you extend the current experiment to match this setting (also re the above convergence point)?*\n\n**Response:** Thanks for this valuable comment! Following your comment, we increased the number of experiment runs to 10 and extended it to 3 step sizes and 1e5 trajectory length. The standard deviation bands do bring down. We have included the detailed results in the revised paper.\n\n**Comment (5):** *In Figure 2, $\\theta_2, \\theta_3, v_2$ have not converged to their true value. What would happen when running for 1e5 time steps? How would the baselines perform at convergence?*\n\n**Response:** We thank the reviewer for the comment. There are two main reasons behind the parameter convergence. From a theoretical perspective, the current model optimizes only the score term and does not jointly optimize the diffusion term $\\sigma_a$, which means the resulting policy may be suboptimal. From an algorithmic standpoint, our method requires the estimation of both the Q-function $Q(x, a)$ and its gradient $\\nabla_a Q$ for policy updates. This dual estimation introduces additional variance and bias, potentially leading to inaccurate policy updates. Notably, these parameters are closely tied to $\\nabla_a Q$. We acknowledge this as an important direction for future work.\n\nIn contrast, the baseline methods only rely on the value function $V(x)$ and are not subject to these issues. Nevertheless, as demonstrated in our experiments, our proposed CQSM method consistently outperforms the baselines in terms of average reward in the early stages of training, achieving higher rewards more quickly. We have now added additional experiments extending the training horizon to $10^5$ time steps with the convergence of these parameters unchanged.\n\n**Comment (6):** *The introduction argues about the necessity for continuous-time RL for real-world applications in autonomous driving, robot manipulation, and high-frequency algorithmic trading. For many of these applications, discrete-time RL has been working well. It would be great to see more extensive experiments that further highlight benefits of CQSM over existing alternatives.*\n\n**Response:**  Thanks for this valuable comment! Our argument is not that discrete-time methods are categorically inadequate, but rather that continuous-time formulations offer theoretical and practical advantages in a random environment at ultra-high frequency. We agree that these potential advantages should be supported with more extensive empirical evidence. In this version, we have evaluated CQSM in more complex, high-dimensional Mujoco tasks (Walker Run and Cheetah Run). The table below shows the average rewards of TD3, SAC and ours. We can see that our method can obtain a larger reward in earlier steps. We have included more detailed results in the final paper.\n|    | Steps| 100k  | 250k  | 500k  | 750k  | 1M    |\n|--------------|--------|-------|-------|-------|-------|-------|\n| **Walker Run** | TD3    | 133   | 266   | 378   | 501   | 592   |\n|              | SAC    | 186   | 496   | 618   | 661   | 681   |\n|              | Ours   | 541   | 678   | 684   | 706   | 727   |\n|              |        |       |       |       |       |       |\n|  | **Steps**  | **100k**  | **150k**   | **200k**  | **250k**   | **300k**   |\n|  **Cheetah Run**   | TD3    | 192   | 302   | 378   | 565   | 683   |\n|              | SAC    | 253   | 559   | 695   | 766   | 800   |\n|              | Ours   | 455   | 628   | 749   | 773   | 832   |\n|              |        |       |       |       |       |       |\n\n**Comment (7):** *It would be nice to provide a summary of the “Policy Gradient” algorithm used as a baseline in order to make the paper more self-contained.*\n\n**Response:** Thanks for this comment! We have added a summary of the “Policy Gradient” algorithm following your comment.\n\n**Comment (8):** *The legend entries of Figure 2 are too small, please enlarge them. Also, the “Time” x-labels appear cut off on my end. line 229, “Then” is capitalized.*\n\n**Response:** Thanks for this suggestion! We have fixed the mentioned figure and typos.\n\n[1] Jia and Zhou. q-learning in continuous time. Journal of Machine Learning Research, 24(161):1–61, 2023.\n\n[2] Doya. Reinforcement learning in continuous time and space. Neural computation, 2000.\n\n[3] Yildiz et al. Continuous-time model-based reinforcement learning. ICML, 2021.\n\n[4] Tavakoli, Arash, et al. Learning to represent action values as a hypergraph on the action vertices. ICLR, 2021.\n\n[5]Seyde, Tim, et al. Solving continuous control via q-learning. ICLR, 2023.\n\n[6] Ireland, David, and Giovanni Montana. Revalued: Regularised ensemble value-decomposition for factorisable markov decision processes. ICLR, 2024."}, {"review_id": "EL6yv4zd7z", "reviewer": "Reviewer_UdxW", "summary": "This paper studies proposes Q-score matching, a continuous-time reinforcement learning algorithm for learning the optimal control of the diffusion process. The paper generalizes the idea of score-matching based diffusion policy to the continuous time and study the extension of the \"Q function\" in discrete time to continuous time such that the dependence on action will not vanish. Empirical evidence is provided for showcasing the effectiveness of the proposed method.", "questions": "1. How can the authors ensure that the parameterization of the diffusion policy will lead to the optimal policy for the original optimal control problem? If I understand correctly, the initial action is also drawn from a non informative prior, which makes the experessiveness policy questionable.", "limitations": "**Application to Broader Tasks**: some references on recent works for applying stochastic control / continuous-time RL for diffusion generative models fine-tuning seem quite related, but are missing: e.g [1][2]. This also raises the questions that whether the proposed algorithm in the paper could be applied to realistic settings: have authors tried to use their proposed algorithms e.g. for diffusion models fine-tuning?\n\n[1] Zhao et. al. Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learning, ICML 2025\n\n[2] Han et.al Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergence, ICML 2025", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strength:**\n1. This paper did a good job in revisiting several relevant studies in the continuous-time RL literature. \n2. The continuous time formulation of the diffusion policies and the resulting definition of Q function seems new and interesting.\n\n**Weakness:**\n1. **Novelty**: proof techniques (e.g. Theorem 1 and Theorem 2) are pretty straightforward, and have already appeared in [18] and [20], thus over theorectical contributions seem quite limited.\n2. **Significance**: The experiments are only based on a low-dimension synthetic stochastic control problem, which is rather limited. No experiments on realistic settings for utilizing realistic application data, which makes the effectiveness of the algorithms not convincing and the broader impact of this paper questionable.\n3. **Typos and Confusing points**: There are several typos and parts that make me confused:\n* On Line 122, what is $\\pi$ here? The definition has not appeared beforehand. \n* On Line 4 of the Algorithm, where does the iterative denoising come from? From an educated guess, some of the notations in this paper are inspired from score-based diffusion models, but there are no relevant discussions and often they just come out from nowhere.", "was_revised": true, "final_justification": "I would like to thank the authors for their detailed responses and efforts in the rebuttal. I have increased my scores due to their hard work. Nevertheless, I still have concerns about several points not well explained in the current version. Hope that these could be addressed in an updated version, which shall also include the authors' additional experiments.", "rebuttal": "We sincerely thank your time and efforts in reviewing our paper and for your valuable comments! We appreciate your positive comment: \"This paper did a good job in revisiting several relevant studies in the continuous-time RL literature, the continuous time formulation of the diffusion policies and the resulting definition of Q function seems new and interesting\". In the following, we provide detailed responses to the comments point by point.\n\n**Comment (1):** *Novelty: proof techniques (e.g. Theorem 1 and Theorem 2) are pretty straightforward, and have already appeared in [18] and [20], thus over theorectical contributions seem quite limited.*\n\n**Response:** We acknowledge that the proof techniques of [18] and [20] have been an important source of inspiration. Our contribution does not lie in introducing new proof methods, but in adapting them to a novel continuous-time, Q-function-based framework that supports score matching without discretization. Theorems 1 and 2 together provide a tractable foundation linking Hamilton-Jacobi-Bellman equation and RL in a new algorithmic context. We have revised the relevant sections to better emphasize this novel contribution.\n\n**Comment (2):** *Significance: The experiments are only based on a low-dimension synthetic stochastic control problem, which is rather limited. No experiments on realistic settings for utilizing realistic application data, which makes the effectiveness of the algorithms not convincing and the broader impact of this paper questionable.*\n\n**Response:** Thanks for this suggestion! We use the LQ setting for its closed-form solution and it is easy to evaluate the convergence of parameters. Note that our method is fully compatible with high-dimensional problems. Its continuous-time and model-free design makes it broadly applicable. We have evaluated the proposed CQSM method in more complex, high-dimensional Mujoco tasks (Walker Run, Cheetah Run). The table below shows the average rewards of TD3, SAC and our method. We can see that our proposed CQSM method can obtain a larger reward in earlier steps. We have included more detailed results in the final paper.\n|    | Steps| 100k  | 250k  | 500k  | 750k  | 1M    |\n|--------------|--------|-------|-------|-------|-------|-------|\n| **Walker Run** | TD3    | 133   | 266   | 378   | 501   | 592   |\n|              | SAC    | 186   | 496   | 618   | 661   | 681   |\n|              | Ours   | 541   | 678   | 684   | 706   | 727   |\n|              |        |       |       |       |       |       |\n|  | **Steps**  | **100k**  | **150k**   | **200k**  | **250k**   | **300k**   |\n|  **Cheetah Run**   | TD3    | 192   | 302   | 378   | 565   | 683   |\n|              | SAC    | 253   | 559   | 695   | 766   | 800   |\n|              | Ours   | 455   | 628   | 749   | 773   | 832   |\n|              |        |       |       |       |       |       |\n\n**Comment (3):** *Typos and Confusing points: There are several typos and parts that make me confused: On Line 122, what is $\\pi$ here? The definition has not appeared beforehand. On Line 4 of the Algorithm, where does the iterative denoising come from? From an educated guess, some of the notations in this paper are inspired from score-based diffusion models, but there are no relevant discussions and often they just come out from nowhere.*\n\n**Response:** Sorry for this confusion! We have clarified that $\\pi$ refers to the action distribution. The iterative denoising step is inspired by score-based diffusion models ([1,2]). The sampling action in our method can use Score matching with Langevin dynamics (SMLD), Denoising diffusion probabilistic modeling (DDPM), or steady-state distribution if there is an analytical solution (details in Line 228-234). We have included this connection and provided additional discussion in the revised paper to avoid confusion. \n\n**Comment (4):** *How can the authors ensure that the parameterization of the diffusion policy will lead to the optimal policy for the original optimal control problem? If I understand correctly, the initial action is also drawn from a non informative prior, which makes the experessiveness policy questionable.*\n\n**Response:** Thank you for the thoughtful question. While the initial action is sampled from a non-informative prior, its influence diminishes as the number of denoising steps $t$ increases. As shown in Line 228 of the paper, as $t\\to \\infty$, the stationary distribution of the diffusion policy depends only on the learned drift $\\Psi$ and diffusion $\\sigma_a$, not the initial action. This allows the policy to be expressive and capable of approximating the optimal solution despite a simple initialization.\n\n**Comment (5):** *Application to Broader Tasks: some references on recent works for applying stochastic control / continuous-time RL for diffusion generative models fine-tuning seem quite related, but are missing: e.g [3,4]. This also raises the questions that whether the proposed algorithm in the paper could be applied to realistic settings: have authors tried to use their proposed algorithms e.g. for diffusion models fine-tuning?* \n\n**Response:** Thank you for the insightful comment. We have cited your mentioned works and clarified the connection between our method and recent diffusion model fine-tuning methods. To our knowledge, existing works (e.q. [5]) view the diffusion-based generative model in a value-function-based framework from a stochastic optimal control perspective. Since our method provides a Q-function-based framework, it could be adapted to diffusion model fine-tuning in a Q-function-based framework, which we consider a promising direction for future work.\n \n[1] Song et al. Score-Based Generative Modeling through Stochastic Differential Equations. ICLR, 2021.\n\n[2] Ho et al. Denoising Diffusion Probabilistic Models. NeurIPS, 2020.\n\n[3]Zhao et. al. Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learning. ICML, 2025.\n\n[4] Han et.al Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergence. ICML, 2025.\n\n[5] Berner et al. An optimal control perspective on diffusion-based generative modeling.\nTransactions on Machine Learning Research, 2024."}, {"review_id": "WBORkqy2Ly", "reviewer": "Reviewer_xNmx", "summary": "This paper proposes Continuous Q‑Score Matching (CQSM), a value‑based RL framework that operates directly in continuous time and targets to tackle stochastic optimal control problems. The authors consider the dynamic programming principle to derive the associated HJB equation for the Q-function and establish a score improvement theorem. In a linear–quadratic (LQ) control task, the authors derive a closed‑form solution of score function and empirically show that CQSM attains higher reward earlier in the training than continuous‑time policy‑gradient and little‑q learning baselines, after which all methods eventually achieve comparable average reward/return.", "questions": "1. Please test multiple orthogonality functions and report their impact on bias and variance.  \n\n2. Provide results on MuJoCo locomotion tasks with larger action spaces and more complicated parameterization of the policy and Q-function.\n\n3. Include diffusion‑RL and more advanced policy gradient algorithm, e.g., DDPG, TD3, SAC etc baselines.  \n\n4. It's a bit weird to put assumptions and theorems in the Experiment section.\n\n5. Since the experiments are very fast to run, the authors can consider increase the number of random seeds to more than 10 for more statistically trust-worthy validation.\n\n6. Why parameterize the Q-function in the form as shown in Line 262? Are there any other parameterization candidates (especially neural networks)? Will they be better? Please try NN and report its performance.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**  \n1. The paper is well-organized and easy to follow with clear intuitions and rigorous derivations.\n\n2. The proposed method is supported by theoretical guarantees and preliminary experimental results.\n\n3. It's a good bridge between diffusion policies and stochastic control.  \n\n4. The LQ problem enables us to measure the exact error between the CQSM-learned parameters and true parameters.\n\n5. The scheme and algorithms are introduced in detail and clearly.\n\n\n\n**Weaknesses**\n\n\n1. The experiments are limited to a naive LQ problem. The scalability of CQSM to high-demensional control tasks such as Mujoco locamotion tasks remains unclear. This is the main weakness.\n\n2. The baselines didn't include modern continuous-time RL algorithms such as DDPG, TD3, SAC etc or diffusion‑RL methods.\n\n3. This paper only tests a simple and low-dimensional parameterization of Q-function and score function. Please see my suggestions below.", "was_revised": true, "final_justification": "The authors' rebuttal has addressed my concerns and answered my questions using additional experiments. They also well-explained the superiority of their method over TD3 and SAC by empirical evidence and theoretical support. I will increase my score.", "rebuttal": "We sincerely thank your time and efforts in reviewing our paper and for your valuable comments! We appreciate  your positive comment: \"the paper is well-organized and easy to follow and the proposed method is supported by theoretical guarantees and preliminary experimental results. It's a good bridge between diffusion policies and stochastic control. The LQ problem enables us to measure the exact error between the CQSM-learned parameters and true parameters. The scheme and algorithms are introduced in detail and clearly\". In the following, we provide detailed responses to the comments point by point.\n\n**Comment (1):** *The experiments are limited to a naive LQ problem. The scalability of CQSM to high-demensional control tasks such as Mujoco locomotion tasks remains unclear. This is the main weakness. Provide results on MuJoCo locomotion tasks with larger action spaces and more complicated parameterization of the policy and Q-function.*\n\n**Response:** Thank you for the comment. We have addressed this by adding experiments on MuJoCo tasks (Walker Run and Cheetah Run) with OpenAI gym (0.23.1) to demonstrate the scalability of CQSM to high-dimensional control settings. The table below shows the average rewards of TD3, SAC and ours. We can see that our method can obtain a larger reward in earlier steps. We have included more detailed results in the final paper.\n|    | Steps| 100k  | 250k  | 500k  | 750k  | 1M    |\n|--------------|--------|-------|-------|-------|-------|-------|\n| **Walker Run** | TD3    | 133   | 266   | 378   | 501   | 592   |\n|              | SAC    | 186   | 496   | 618   | 661   | 681   |\n|              | Ours   | 541   | 678   | 684   | 706   | 727   |\n|              |        |       |       |       |       |       |\n|  | **Steps**  | **100k**  | **150k**   | **200k**  | **250k**   | **300k**   |\n|  **Cheetah Run**   | TD3    | 192   | 302   | 378   | 565   | 683   |\n|              | SAC    | 253   | 559   | 695   | 766   | 800   |\n|              | Ours   | 455   | 628   | 749   | 773   | 832   |\n|              |        |       |       |       |       |       |\n\n\n**Comment (2):**  *The baselines didn't include modern continuous-time RL algorithms such as DDPG, TD3, SAC etc or diffusion‑RL methods.*\n\n**Response:** Thank you for the suggestion. In the added MuJoCo experiments, we now include comparisons with modern continuous-time RL baselines such as TD3 and SAC. In addition, the two baseline models (PG and q-learning) in the paper are the actor–critic learning in continuous time and space. Furthermore, Jia et al. (JMLR 2023) [1] have verified that the discrete Q-learning algorithm is noisier and slower in convergence speed compared with their proposed continuous time algorithms, from both theoretical and experimental perspectives. We have revised the Experiments section to provide a clearer comparison.\n\n\n**Comment (3):** *Please test multiple orthogonality functions and report their impact on bias and variance.*\n\n**Response:** Thank you for the suggestion. For the choice of the parametric family $Q^{\\theta}$, general choices include linear combinations of some basis functions (LQ) or neural networks (MuJoCo experiments). However, analytical solutions are only available under quadratic Q-function. Replacing it with other orthogonal bases (e.g., Legendre, RBF) would no longer preserve the LQ structure, making it infeasible to compute the ground-truth bias. For common choices of the test functions are $\\xi_t=\\frac{\\partial Q^{\\theta}}{\\partial \\theta}$ or $\\xi_t=\\int_0^t \\rho^{s-t}\\frac{\\partial Q^{\\theta}}{\\partial \\theta} ds, 0<\\rho\\leq 1.$  We conduct experiments using different testing functions $\\xi_t=\\int_0^t \\rho^{s-t}\\frac{\\partial Q^{\\theta}}{\\partial \\theta} ds$ with $\\rho=0.1,0.5,1$. The results are similar to those obtained using the test function $\\xi_t=\\frac{\\partial Q^{\\theta}}{\\partial \\theta}$, but the algorithm runs more slowly.\n\n\n**Comment (4):** *It's a bit weird to put assumptions and theorems in the Experiment section.*\n\n**Response:** Thank you for pointing this out. We have moved assumptions and theorems to the Appendix to improve the logical flow and clarity of the main text.\n\n**Comment (5):** *Since the experiments are very fast to run, the authors can consider increase the number of random seeds to more than 10 for more statistically trust-worthy validation.*\n\n**Response:** Thank you! We have increased the number of experiment runs to 10 and revised the results in the paper.\n\n**Comment (6):** *Why parameterize the Q-function in the form as shown in Line 262? Are there any other parameterization candidates (especially neural networks)? Will they be better? Please try NN and report its performance.*\n\n**Response:** Thanks for the comment! We use the quadratic parameterization in Line 262 because this is the form of the theoretical solution obtained based on the previous LQ problem which admits a closed-form solution, enabling precise evaluation of bias and variance. In Mujoco experiments, we do employ neural network parameterizations. The average reward performance of CQSM is better than SAC and TD3 in earlier steps. \n\n[1] Jia and Zhou. q-learning in continuous time. Journal of Machine Learning Research, 24(161):1–61, 2023."}, {"review_id": "Zp7cif2Xnp", "reviewer": "Reviewer_v8YX", "summary": "This paper introduce a RL method for continuous-time control problems with diffusion policies. The authors derive a Bellman equation for continuous-time Q functions and propose a score improvement principle. The perspective of continuous-time Q-learning is theoretically valuable. The paper is well-organized and with clearly assumptions and proofs.", "questions": "1. Is the proposed method able to degenerated from a stochastic version into a deterministic version? If it is true, I suggest that the comparison experiment should be further conducted.\n2. What is the intuitionistic meaning of operator $\\mathcal{L}$? There should be some explanation or references.\n3. In section 4, the authors mentioned the proposed approach avoids “some of the challenges” inherent to policy gradient methods. What is the specific challenges?\n4. In performance results, the paths of learned $\\theta_0, \\theta_1,\\theta_2,\\theta_4,\\theta_5$, and $v_0,v_1,v_2$  are not converged to the theoretical value. Please explain the reason why the parameters can not converge for even a linear system.\n5. Is the method able to extended into a nonlinear version? An experiment on complex and nonlinear systems will make the proposed method more persuasive.\n6. Fig3 does not use vector graphics, the image quality is very low, and the font is very small.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. The framework learns the Q-function directly in continuous time via a martingale condition, thus preserving action sensitivity and avoiding the bias/degeneration that often appears when time is forcibly discretised.\n2. The authors prove that the diffusion-policy score field Ψ equals λ⁻¹ ∇ₐ Q at optimum and establish a Score-Improvement Theorem.\n\nWeakness:\n1. The authors mention that there are some recent work in continuous Q-learning in deterministic dynamics ([8], [33], [18], [19], [20]). However, there is no comparison between the proposed method and these mentioned methods in the experiment section.\n2. The definition of operator $\\mathcal{L}$ is abrupt and without any references.\n3. The Hamilton-Jacobi-Bellman equation (eq.6) is not in a standard form. Please elaborate the relationship between eq.6 and the standard form and add more references.\n4. Related work is not comprehensive. Papers related to QSM, such as SDAC(Soft Diffusion Actor-Critic: Efficient Online Reinforcement Learning for Diffusion Policy) and DACER(Diffusion actor-critic with entropy regulator), are not included in the main text.", "was_revised": true, "final_justification": "The theory of this paper is very solid, but the experimental part still has some shortcomings, especially the lack of the latest algorithms.", "rebuttal": "We sincerely thank your time and efforts in reviewing our paper and for your valuable comments! We appreciate  your positive comment: \"The framework learns the Q-function directly in continuous time preserving action sensitivity and avoiding the bias/degeneration that often appears when time is forcibly discretized. The diffusion-policy score field $\\Psi=\\nabla_a Q$ at optimum have been proved and a Score-Improvement Theorem has been established\". In the following, we provide detailed responses to the comments point by point.\n\n**Comment (1):** *The authors mention that there are some recent work in continuous Q-learning in deterministic dynamics ([8], [33], [18], [19], [20]). However, there is no comparison between the proposed method and these mentioned methods in the experiment section.*\n\n**Response:** Thank you for the suggestion. We would like to clarify that [18], [19], and [20] correspond to the two baseline methods already included in our experiments: Policy Gradient (PG) and little-q learning. Gao et al. (SICON 2022) [8] focus on entropy-based exploration for non-convex optimization, rather than RL. Meanwhile, [33] provides a theoretical analysis showing that Q-functions degenerate into value functions in continuous time, which serves as a foundation for [20]. In [20], it is further shown that the algorithm based on [33] (discrete Q-learning) exhibits higher variance and slower convergence compared to little-q learning. We have revised the manuscript to clarify these distinctions.\n\n\n**Comment (2):** *The definition of operator $\\mathcal{L}$ is abrupt and without any references. What is the intuitionistic meaning of operator $\\mathcal{L}$?*\n\n**Response:** Thank you for pointing this out. We have now added an explanation that the operator $\\mathcal{L}$ represents the infinitesimal operator of the joint state-action diffusion process $(X_t, a_t)$, which plays a fundamental role in continuous-time stochastic control theory. Its form can be derived using standard techniques analogous to those used for the value function $V(x)$, and for brevity, we omit the detailed derivation. We have clarified the definition of $\\mathcal{L}$ in the main text and included a reference to standard results in stochastic control (e.g., [1, Chapter 4]).\n\n\n\n**Comment (3):** *The Hamilton-Jacobi-Bellman equation (eq.6) is not in a standard form. Please elaborate the relationship between eq.6 and the standard form and add more references.*\n\n**Response:** Thank you for the comment. Eq.6 is indeed a non-standard form of the HJB equation, as it is written in terms of the Q-function rather than the value function. This formulation is motivated by our goal of developing a Q-learning-type algorithm in continuous time. The equation can be viewed as an alternative characterization of the standard Hamilton-Jacobi-Bellman equation through a state-action function, where the minimization is carried out over scores of action processes. We have now added a clarifying remark and references (e.g., [1], [21]) to explain this relationship more explicitly.\n\n\n**Comment (4):** *Related work is not comprehensive. Papers related to QSM, such as SDAC(Soft Diffusion Actor-Critic: Efficient Online Reinforcement Learning for Diffusion Policy) and DACER(Diffusion actor-critic with entropy regulator), are not included in the main text.* \n\n**Response:** Thank you! We have extended the Related Work section including the papers you mentioned.\n\n\n**Comment (5):** *Is the proposed method able to degenerated from a stochastic version into a deterministic version? If it is true, I suggest that the comparison experiment should be further conducted.\"*\n\n**Response:** Yes, our method naturally reduces to a deterministic version without the diffusion term. In the experiment section, we set $C = D = 0$ to eliminate stochasticity and observe that the resulting performance is comparable to the stochastic case. We have included this comparison experiment in the revised paper.\n\n**Comment (6):** *In section 4, the authors mentioned the proposed approach avoids “some of the challenges” inherent to policy gradient methods. What is the specific challenges?* \n\n**Response:** Thank you for the comment. The challenges include: (i) discrete-time policy gradient algorithms often suffer from sample inefficiency due to backpropagation through the entire diffusion chain [27], and tend to be noisier [20]; (ii) existing continuous-time policy gradient methods [19] typically require an additional policy evaluation step, which complicates implementation. Our method avoids these issues by directly optimizing a continuous-time Q-function through score matching. We have clarified this point in the revised Section 4.\n\n**Comment (7):** *In performance results, the paths of learned $\\theta_1,\\theta_2,\\theta_3,\\theta_4,\\theta_5$, and $v_1,v_2,v_3$ are not converged to the theoretical value. Please explain the reason why the parameters can not converge for even a linear system.*\n\n**Response:** Thank you! There are two main reasons behind the mismatch. From a theoretical perspective, the current model optimizes only the score term and does not jointly optimize the diffusion term $\\sigma_a$, which means the resulting policy may be suboptimal. From an algorithmic standpoint, our CQSM framework requires the estimation of both the Q-function $Q(x, a)$ and its gradient $\\nabla_a Q$ for policy updates. This dual estimation introduces additional variance and bias, potentially leading to inaccurate policy updates. Notably, the parameters $\\theta_2,\\theta_3,v_2$ are closely tied to $\\nabla_a Q$, while the other parameters are well converged to the theoretical value. We acknowledge this as an important direction for future work. However, as shown in our experiments, the proposed CQSM method still outperforms baselines in terms of average reward in the early stages of training, achieving higher rewards more quickly. \n\n\n**Comment (8):** *Is the method able to extended into a nonlinear version? An experiment on complex and nonlinear systems will make the proposed method more persuasive.*\n\n**Response:** Thank you for the suggestion. In principle, our framework is extendable to nonlinear systems by parameterizing the Q-function and policy with expressive function approximators such as neural networks. We have evaluated our method in more complex, high-dimensional Mujoco tasks (Walker Run, Cheetah Run). The table below shows the average rewards of TD3, SAC, and ours. We can see that our method can obtain a larger reward in earlier steps. We have included more detailed results in the final paper.\n|    | Steps| 100k  | 250k  | 500k  | 750k  | 1M    |\n|--------------|--------|-------|-------|-------|-------|-------|\n| **Walker Run** | TD3    | 133   | 266   | 378   | 501   | 592   |\n|              | SAC    | 186   | 496   | 618   | 661   | 681   |\n|              | Ours   | 541   | 678   | 684   | 706   | 727   |\n|              |        |       |       |       |       |       |\n|  | **Steps**  | **100k**  | **150k**   | **200k**  | **250k**   | **300k**   |\n|  **Cheetah Run**   | TD3    | 192   | 302   | 378   | 565   | 683   |\n|              | SAC    | 253   | 559   | 695   | 766   | 800   |\n|              | Ours   | 455   | 628   | 749   | 773   | 832   |\n|              |        |       |       |       |       |       |\n\n**Comment (9):** *Fig3 does not use vector graphics, the image quality is very low, and the font is very small.* \n\n**Response:** Thanks! We have fixed the mentioned figure.\n\n\n[1] Yong, Jiongmin and Zhou. Stochastic controls: Hamiltonian systems and HJB equations.\n  Springer Science \\& Business Media, 1999.\n\n[8] Gao et al. State-dependent temperature control for langevin diffusions. SIAM Journal on Control and Optimization, 2022.\n\n[18] Jia and Zhou. Policy evaluation and temporal-difference learning in continuous\ntime and space: A martingale approach. Journal of Machine Learning Research, 2022.\n\n[19] Jia and Zhou. Policy gradient and actor-critic learning in continuous time and space: Theory and algorithms. Journal of Machine Learning Research, 2022.\n\n[20] Jia and Zhou. q-learning in continuous time. Journal of Machine Learning\nResearch, 2023.\n\n[21] Kim et al. Hamilton-jacobi deep q-learning for deterministic\ncontinuous-time systems with lipschitz continuous controls. Journal of Machine Learning Research, 2021.\n\n[27] Psenka et al. Learning a diffusion model policy from rewards via q-score matching. In International Conference on Machine Learning, 2024.\n\n[33] Tallec et al. Making deep q-learning methods robust to time discretization. In International Conference on Machine Learning, 2019."}]

Author rebuttal:

Meta-review: Referees (and my own reading) agree that the paper presents an interesting and genuinely novel framework, supported by theory, and having an interesting specialization to the LQ case. Furthermore, they state the paper is well-written.

While some concerns (like existing proof techniques) are acknowledged by the authors, their rebuttal does manage to convince the referees that the positives outweigh the concerns for this submission. Other concerns are well addressed with novel experiments; these include the simplicity of the task/scalability to more realistic and fast tasks that more strongly benefit from the continuous-time formalism; and unconvincing baselines in the initial version of the paper.

Overall, I recommend acceptance for this paper.
Decision: Accept (poster)