Title: Value Diffusion Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Value Diffusion learning; Distributional Reinforcement Learning
Abstract: Model-free reinforcement learning (RL) combined with diffusion models has achieved significant progress in addressing complex continuous control tasks. However, a persistent challenge in RL remains the accurate estimation of Q-values, which critically governs the efficacy of policy optimization. Although recent advances employ parametric distributions to model value distributions for enhanced estimation accuracy, current methodologies predominantly rely on unimodal Gaussian assumptions or quantile representations. These constraints introduce distributional bias between the learned and true value distributions, particularly in some tasks with a nonstationary policy, ultimately degrading performance. To address these limitations, we propose value diffusion reinforcement learning (VDRL), a novel model-free online RL method that utilizes the generative capacity of diffusion models to represent multimodal value distributions. The core innovation of VDRL lies in the use of the variational loss of diffusion-based value distribution, which is theoretically proven to be a tight lower bound for the optimization objective under the KL-divergence measurement. Furthermore, we introduce double value diffusion learning with sample selection to enhance training stability and further improve value estimation accuracy. Extensive experiments conducted on the MuJoCo benchmark demonstrate that VDRL significantly outperforms some SOTA model-free online RL baselines, showcasing its effectiveness and robustness.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Value Diffusion Reinforcement Learning (VDRL), a model-free, off-policy reinforcement learning algorithm that employs diffusion models to represent value distributions. To further improve stability, they propose double value diffusion learning with sample selection, inspired by double Q-learning. VDRL is evaluated on eight MuJoCo benchmark tasks and compared against eleven baseline RL methods spanning traditional, distributional, and diffusion-policy RL paradigms.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Addressing Q-value estimation bias is a core challenge in RL. VDRL has the potential to significantly impact future work in distributional RL.\n- Value estimation bias is quantitatively analyzed, which is an uncommon but insightful inclusion.\n- The paper makes a compelling case for moving beyond unimodal Gaussian value approximations, providing both empirical and conceptual justification.\n\nWeaknesses:\n- The proof in Line 500 relies on the assumption that the policy has fully converged (the authors do not mention this in Theorem 1). However, in practice, the policy is extremely hard to fully converge, especially when using a neural network as the policy. Therefore, $\\frac{P_{\\theta}(Y_{1:T} \\mid s, a, Y_0)}{q(Y_{1:T} \\mid s, a, Y_0)}=1$ in Line 500 does not hold in general, which means Theorem 1 does not hold.\n- There are no concrete data for training and inference time. The training and inference processes of diffusion models are notably time-consuming. Whether the performance improvements are worth such a high computational cost is debatable.\n- The x-axis in Figure 1 should be changed from training steps to wall time; otherwise, the comparison is unfair because diffusion models need more time for training in general.\n- Line 40 mentions quantile-based methods [27–32], which are also multimodal value distributions, but no experiments are conducted to compare against these baselines.\n\nMoreover, I just noticed a paper [1] that has already been accepted but is not cited in your work. Its background and method are highly similar to yours. Both use diffusion models to fit multimodal value networks, and the loss functions are identical—i.e., Equation (10) in your paper and Equation (15) in [1]. Could you clarify the differences between your work and this accepted paper?\n\n[1] Tong Liu, et al. Distributional Soft Actor-Critic with Diffusion Policy. ITSC 2025. ([arXiv](https://arxiv.org/abs/2507.01381))\n\nQUESTIONS:\n- Can you provide wall-clock training and inference times per step for VDRL compared to baselines?\n- After changing the x-axis of Figure 1 to wall time, does VDRL still retain its advantage?\n- How does VDRL compare to the quantile-based methods [27–32]?\n- I noticed that the performance of your DACER is quite different from that in the original paper. The distributed Q learning part based on DACER is the same as DSACT, but the performance in the paper is lower than DSACT. Can you run the experiment using the jax version of the code?\n- What are the differences between your work and the accepted paper (Liu et al, ITSC 2025)?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-23"}

comments: SUMMARY:
This paper aims to address the distributional bias in value distributions via diffusion models (multi-modal value distributions). Specifically, the proposed variational loss of diffusion-based value distribution is theoretically proven to be the lower bound of the optimization objective.
Experimental results on the MuJoCo benchmark demonstrate that the proposed VDRL outperforms existing baselines significantly.

STRENGTHS AND WEAKNESSES:
### Strengths
1. It is an interesting idea to integrate the diffusion model and distributional RL.
2. The writing is clear, but the structure should be improved.

### Weakness
1. Lines 171-174: The statements of underestimation and overestimation are not correct. These estimation biases have different effects in different task settings [1]. Additionally, it is possible to control the estimation bias via maxmin value estimation.
2. There are no related experiments of value estimation accuracy that validate the paper's statements: "utilizing the generative capacity of diffusion models to represent expressive and multimodal value distribution"


### Minor
1. For coherency of reading, the related work section is usually placed after the introduction section or before the conclusion section.

[1] Lan, Qingfeng, et al. "Maxmin q-learning: Controlling the estimation bias of q-learning." arXiv preprint arXiv:2002.06487 (2020).

QUESTIONS:
### Questions
1. Could the authors provide the training and inference speed comparison? One of my concerns is the computational budget of the diffusion value function.
2. Experimental results on value estimation accuracy.
3. Experimental results on discrete control tasks (e.g., Atari-10). Comparison of the proposed diffusion value method with the IQN.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Value Diffusion Reinforcement Learning (VDRL) which employs diffusion models to learn expressive and multimodal returns in an off-policy distributional RL framework. The authors prove that a diffusion variational lower bound (VLB) is a tight lower bound on the KL divergence between the Bellman-updated target distribution and the learned diffusion critic (Theorem 1). They then build an algorithm incorporating mainstream RL techniques—namely a double-critic sampling method—to mitigate overestimation and stabilize training. Experimental results on MuJoCo locomotion tasks provide some statistical evidence that VDRL consistently outperforms baselines (PPO, TD3, SAC, DSAC, DSACT, and diffusion-based methods DIPO, QSM, QVPO, DACER) in both mean return and Q-estimation bias.\n\nSTRENGTHS AND WEAKNESSES:\n1. Quality\n    * Strength: Theorem 1 cleanly links diffusion VLB to the RL KL-loss and gives a concrete algorithmic foundation.\n    * Weakness: Theorem 1 follows directly from standard DDPM variational bounds (e.g., Eq. 5 in ref [41]), thus offering limited new insight. As part of their theoretical exploration it would have been nice to see an analysis of Bellman-operator contraction/convergence (like in C51 [Bellemare et al. 2017], IQN [Dabney et al. 2018]), although this is not strictly necessary as long as learning curves show stability and convergence (which they do), but this brings me to the next point which is the very low number of seeds (4) used to make statistical arguments of the superiority of the method. As a rule of thumb, having 10 seeds is a requirement to make such statement; any less would be questionable so I would not take their empirical tests seriously unless they fix it in the rebuttal. I looked at one of the main baselines (QVPO) [1], and there doesn’t seems to be consistent final returns between the two papers of the algorithm QVPO judging by Figure 1 in your paper and Figure 3 in [1].\n2. Clarity\n    * Strength: Well-motivated exposition with clear pseudocode (Algorithm 1).\n    * Weakness: Minor typos (e.g., “introducs”, “alogirthms”); Eq. 10 needs a revision. Hyperparameters (diffusion steps, noise schedules, seed count) are scattered across the main text, Section 5.4, and Appendix D instead of centralized.\n3. Significance\n    * Strength: Addresses the limitation of unimodal Gaussian or quantile critics by modeling full return distributions, leading to better representation of complex reward landscapes.\n    * Weakness: Evaluation is confined to deterministic MuJoCo environments; lacks tests under sensor/motor noise, partial observability, or on real-world robotic platforms. The paper also omits reporting compute–performance tradeoffs (e.g., reverse-sampling cost, wall-clock metrics).\n4. Originality\n    * Strength: First to apply diffusion generative modeling directly to value distributions in RL rather than to policies.\n    * Weakness: The work overlaps heavily with recent diffusion-value works (Mazoure et al. 2023; QSM; QVPO). The core DDPM VLB machinery is reused rather than extended, so the conceptual novelty is incremental.\n\n\nRef:\n[1] Ding, S. et al . Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization. NeurIPS 2024.\n\nQUESTIONS:\n1. Do MuJoCo locomotion tasks exhibit true multimodal return distributions? (any refs if yes)? Have you considered inherently multimodal benchmarks (e.g., Atari with stochastic rewards, hierarchical tasks, synthetic bimodal MDPs) to directly test your critic’s multimodal modeling claims? \n2. How does VDRL’s objective, derivation, and performance differ from QSM and QVPO? Please provide theoretical distinctions and empirical comparisons on a common benchmark.\n3. Could you consider reporting the per-iteration wall-clock time and total GPU-hours for VDRL vs. DSAC/DSACT.\n4. Will you consider testing VDRL under observation/action noise? If not, discuss anticipated failure modes and possible mitigation strategies.\n\nLIMITATIONS:\n* Compute Latency: Appendix E notes computational overhead but fails to provide concrete latency or throughput metrics, which are critical for high-frequency control applications.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAccurate Q-value estimation is essential for effective policy optimization in reinforcement learning. This work introduces Value Diffusion Reinforcement Learning (VDRL), a model-free online RL approach that uses diffusion models to capture rich, multimodal value distributions. The key idea is a diffusion-based variational loss, theoretically shown to provide a tight lower bound under KL-divergence, leading to more precise value estimation. To further stabilize training, the authors propose a double value diffusion mechanism with sample selection. Experiments on MuJoCo benchmarks show that VDRL consistently outperforms strong state-of-the-art baselines, demonstrating its effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\nThe paper tackles an important problem, which is using diffusion models for value estimation in RL.\n\nThe method is motivated by theoretical analysis and results.\n\nThe experimental evaluation is rich and well executed comparing against 11 baseline algorithms on 8 tasks from Mujococ, reporting both plots and tables.\n\nThe paper is well-organized and well-written.\n\n**Weaknesses:**\n\nThe computational overhead of using diffusion in value estimation is not thoroughly analyzed. There is only a brief mention of trade-offs in the sensitivity analysis (Section 5.4) as well as a brief mention in the limitations of the computational overhead during online training, particularly in high-dimensional action space (Appendix E). Runtime analysis and computational complexity analysis are lacking.\n\nWhile MuJoCo is a representative, standard, and popular benchmark, it remains unclear how VDRL would perform in other tasks.\n\nSensitivity analysis was only conducted on Humanoid-v3 task as an example. However, it remains unclear if the same conclusions holds on the other tasks. The authors should provide ablations on different environments (say three of them) to confirm the observation.\n\nQUESTIONS:\n(1) Can the authors provide detailed runtime comparisons compared to the other baselines? \n\n(2) Can the authors expand the sensitivity analysis to at least two or three additional environments to confirm the consistency of conclusions?\n\n(3) How critical is the double value diffusion mechanism? Could the authors provide an ablation showing performance with and without this?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}