Title: Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Control
Venue: NeurIPS 2025
Accepted: yes
Keywords: Stochastic Control; Reinforcement Learning; Diffusion Policy; Continuous Q-Learning; Martingale Orthogonality Condition
Abstract: Reinforcement learning (RL) has achieved significant success across a wide range of domains, however, most existing methods are formulated in discrete time. In this work, we introduce a novel RL method for continuous-time control, where stochastic differential equations govern state-action dynamics. Departing from traditional value function-based approaches, our key contribution is the characterization of continuous-time Q-functions via a martingale condition and the linking of diffusion policy scores to the action gradient of a learned continuous Q-function by the dynamic programming principle. This insight motivates Continuous Q-Score Matching (CQSM), a score-based policy improvement algorithm. Notably, our method addresses a long-standing challenge in continuous-time RL: preserving the action-evaluation capability of Q-functions without relying on time discretization. We further provide theoretical closed-form solutions for linear-quadratic (LQ) control problems within our framework. Numerical results in simulated environments demonstrate the effectiveness of our proposed method and compare it to popular baselines.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduce a RL method for continuous-time control problems with diffusion policies. The authors derive a Bellman equation for continuous-time Q functions and propose a score improvement principle. The perspective of continuous-time Q-learning is theoretically valuable. The paper is well-organized and with clearly assumptions and proofs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The framework learns the Q-function directly in continuous time via a martingale condition, thus preserving action sensitivity and avoiding the bias/degeneration that often appears when time is forcibly discretised.\n2. The authors prove that the diffusion-policy score field Ψ equals λ⁻¹ ∇ₐ Q at optimum and establish a Score-Improvement Theorem.\n\nWeakness:\n1. The authors mention that there are some recent work in continuous Q-learning in deterministic dynamics ([8], [33], [18], [19], [20]). However, there is no comparison between the proposed method and these mentioned methods in the experiment section.\n2. The definition of operator $\\mathcal{L}$ is abrupt and without any references.\n3. The Hamilton-Jacobi-Bellman equation (eq.6) is not in a standard form. Please elaborate the relationship between eq.6 and the standard form and add more references.\n4. Related work is not comprehensive. Papers related to QSM, such as SDAC(Soft Diffusion Actor-Critic: Efficient Online Reinforcement Learning for Diffusion Policy) and DACER(Diffusion actor-critic with entropy regulator), are not included in the main text.\n\nQUESTIONS:\n1. Is the proposed method able to degenerated from a stochastic version into a deterministic version? If it is true, I suggest that the comparison experiment should be further conducted.\n2. What is the intuitionistic meaning of operator $\\mathcal{L}$? There should be some explanation or references.\n3. In section 4, the authors mentioned the proposed approach avoids “some of the challenges” inherent to policy gradient methods. What is the specific challenges?\n4. In performance results, the paths of learned $\\theta_0, \\theta_1,\\theta_2,\\theta_4,\\theta_5$, and $v_0,v_1,v_2$  are not converged to the theoretical value. Please explain the reason why the parameters can not converge for even a linear system.\n5. Is the method able to extended into a nonlinear version? An experiment on complex and nonlinear systems will make the proposed method more persuasive.\n6. Fig3 does not use vector graphics, the image quality is very low, and the font is very small.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Continuous Q‑Score Matching (CQSM), a value‑based RL framework that operates directly in continuous time and targets to tackle stochastic optimal control problems. The authors consider the dynamic programming principle to derive the associated HJB equation for the Q-function and establish a score improvement theorem. In a linear–quadratic (LQ) control task, the authors derive a closed‑form solution of score function and empirically show that CQSM attains higher reward earlier in the training than continuous‑time policy‑gradient and little‑q learning baselines, after which all methods eventually achieve comparable average reward/return.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**  \n1. The paper is well-organized and easy to follow with clear intuitions and rigorous derivations.\n\n2. The proposed method is supported by theoretical guarantees and preliminary experimental results.\n\n3. It's a good bridge between diffusion policies and stochastic control.  \n\n4. The LQ problem enables us to measure the exact error between the CQSM-learned parameters and true parameters.\n\n5. The scheme and algorithms are introduced in detail and clearly.\n\n\n\n**Weaknesses**\n\n\n1. The experiments are limited to a naive LQ problem. The scalability of CQSM to high-demensional control tasks such as Mujoco locamotion tasks remains unclear. This is the main weakness.\n\n2. The baselines didn't include modern continuous-time RL algorithms such as DDPG, TD3, SAC etc or diffusion‑RL methods.\n\n3. This paper only tests a simple and low-dimensional parameterization of Q-function and score function. Please see my suggestions below.\n\nQUESTIONS:\n1. Please test multiple orthogonality functions and report their impact on bias and variance.  \n\n2. Provide results on MuJoCo locomotion tasks with larger action spaces and more complicated parameterization of the policy and Q-function.\n\n3. Include diffusion‑RL and more advanced policy gradient algorithm, e.g., DDPG, TD3, SAC etc baselines.  \n\n4. It's a bit weird to put assumptions and theorems in the Experiment section.\n\n5. Since the experiments are very fast to run, the authors can consider increase the number of random seeds to more than 10 for more statistically trust-worthy validation.\n\n6. Why parameterize the Q-function in the form as shown in Line 262? Are there any other parameterization candidates (especially neural networks)? Will they be better? Please try NN and report its performance.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies proposes Q-score matching, a continuous-time reinforcement learning algorithm for learning the optimal control of the diffusion process. The paper generalizes the idea of score-matching based diffusion policy to the continuous time and study the extension of the \"Q function\" in discrete time to continuous time such that the dependence on action will not vanish. Empirical evidence is provided for showcasing the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength:**\n1. This paper did a good job in revisiting several relevant studies in the continuous-time RL literature. \n2. The continuous time formulation of the diffusion policies and the resulting definition of Q function seems new and interesting.\n\n**Weakness:**\n1. **Novelty**: proof techniques (e.g. Theorem 1 and Theorem 2) are pretty straightforward, and have already appeared in [18] and [20], thus over theorectical contributions seem quite limited.\n2. **Significance**: The experiments are only based on a low-dimension synthetic stochastic control problem, which is rather limited. No experiments on realistic settings for utilizing realistic application data, which makes the effectiveness of the algorithms not convincing and the broader impact of this paper questionable.\n3. **Typos and Confusing points**: There are several typos and parts that make me confused:\n* On Line 122, what is $\\pi$ here? The definition has not appeared beforehand. \n* On Line 4 of the Algorithm, where does the iterative denoising come from? From an educated guess, some of the notations in this paper are inspired from score-based diffusion models, but there are no relevant discussions and often they just come out from nowhere.\n\nQUESTIONS:\n1. How can the authors ensure that the parameterization of the diffusion policy will lead to the optimal policy for the original optimal control problem? If I understand correctly, the initial action is also drawn from a non informative prior, which makes the experessiveness policy questionable.\n\nLIMITATIONS:\n**Application to Broader Tasks**: some references on recent works for applying stochastic control / continuous-time RL for diffusion generative models fine-tuning seem quite related, but are missing: e.g [1][2]. This also raises the questions that whether the proposed algorithm in the paper could be applied to realistic settings: have authors tried to use their proposed algorithms e.g. for diffusion models fine-tuning?\n\n[1] Zhao et. al. Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learning, ICML 2025\n\n[2] Han et.al Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergence, ICML 2025", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Continuous Q-Score Matching, a model-free reinforcement learning algorithm that leverages diffusion score functions to facilitate learning Q-functions in continuous-time settings. The approach is validated on a class of linear quadratic control problems, analyzing convergence to the analytic parameter optima as well as comparing performance against a selection of baselines.\n\nSTRENGTHS AND WEAKNESSES:\n- The paper is overall well-written and formatted, the manuscript is quite notation heavy and there is a bit of a disconnect between the presented motivation and the experimental validation\n- Line 20-22: low-level robot control systems often run at 0.1-1.0kHz, which is quite fine-grained. Could you elaborate on which dynamics are poorly captured using discretization?\n- Line 32: there is a line of work on discretizing continuous action spaces to apply Q-learning in  high-dimensional “continuous control” settings (e.g., [1-3])\n- If each experiments only takes 4.5 seconds, why not increase the number of runs from 5 to 10 or more to bring down some of the standard deviation bands?\n- In Figure 2, \\theta_2, \\theta_3, \\nu_2 have not converged to their true value. What would happen when running for 1e5 time steps? How would the baselines perform at convergence?\n- The introduction argues about the necessity for continuous-time RL for real-world applications in autonomous driving, robot manipulation, and high-frequency algorithmic trading. For many of these applications, discrete-time RL has been working well. It would be great to see more extensive experiments that further highlight benefits of CQSM over existing alternatives.\n- Furthermore, the experiments in q-learning [4] evaluated 3 step sizes, for 100 re-runs, and 1e5 trajectory length - can you extend the current experiment to match this setting (also re the above convergence point)?\n- It would be nice to provide a summary of the “Policy Gradient” algorithm used as a baseline in order to make the paper more self-contained\n- The legend entries of Figure 2 are too small, please enlarge them. Also, the “Time” x-labels appear cut off on my end.\n- Minor: line 229, “Then” is capitalized\n\n**References:**\n\n[1] Tavakoli, Arash, et al. \"Learning to represent action values as a hypergraph on the action vertices.\" ICLR, 2021.\n\n[2] Seyde, Tim, et al. \"Solving continuous control via q-learning.\" ICLR, 2023.\n\n[3] Ireland, David, and Giovanni Montana. \"Revalued: Regularised ensemble value-decomposition for factorisable markov decision processes.\" ICLR, 2024.\n\n[4] Yanwei Jia and Xun Yu Zhou. q-learning in continuous time. Journal of Machine Learning 346 Research, 24(161):1–61, 2023.\n\nQUESTIONS:\nPlease refer to the \"Strengths and Weaknesses\" section, with the main suggestion relating to more extensive experimental validation.\n\nLIMITATIONS:\nLimitations do not appear to be explicitly addressed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-05"}