Value Diffusion Reinforcement Learning
Authors: Xiaoliang Hu
Fuyun Wang
Tong Zhang
Zhen Cui
Venue: NeurIPS 2025 poster ()
Keywords: Value Diffusion learning; Distributional Reinforcement Learning
Primary area: reinforcement_learning

Abstract: Model-free reinforcement learning (RL) combined with diffusion models has achieved significant progress in addressing complex continuous control tasks. However, a persistent challenge in RL remains the accurate estimation of Q-values, which critically governs the efficacy of policy optimization. Although recent advances employ parametric distributions to model value distributions for enhanced estimation accuracy, current methodologies predominantly rely on unimodal Gaussian assumptions or quantile representations. These constraints introduce distributional bias between the learned and true value distributions, particularly in some tasks with a nonstationary policy, ultimately degrading performance. To address these limitations, we propose value diffusion reinforcement learning (VDRL), a novel model-free online RL method that utilizes the generative capacity of diffusion models to represent multimodal value distributions. The core innovation of VDRL lies in the use of the variational loss of diffusion-based value distribution, which is theoretically proven to be a tight lower bound for the optimization objective under the KL-divergence measurement. Furthermore, we introduce double value diffusion learning with sample selection to enhance training stability and further improve value estimation accuracy. Extensive experiments conducted on the MuJoCo benchmark demonstrate that VDRL significantly outperforms some SOTA model-free online RL baselines, showcasing its effectiveness and robustness.

Value Diffusion Reinforcement Learning
Xiaoliang Hu' Fuyun Wang' Tong Zhang Zhen Cui
School of Computer Science and Engineering, Nanjing University Science and Technology School of Artificial Intelligence. Beijing Normal University ppeter_hu_xle fy 1yw278828)ojjust edu..cduuccom cn
Abstract
Model-free reinforcement learning (RL) combined with diffusion models has achieved significant progress addressing complex continuous control tasks. How- ever, a persistent challenge in RL remains the accurate estimation of Q-values, which critically governs efficacy of policy optimization. Although recent ad- vances employ parametric distributions model value distributions for enhanced estimation accuracy, current methodologies predominantly rely on unimodal Gaus- sian assumptions or quantile representations. These constraints introduce distribu- tional bias between the learned and true value distributions, particularly in some tasks with nonstationary policy, ultimately degrading performance. To address these limitations, we propose value diffusion reinforcement learning (VDRL), a novel model-free ffree online RL method that utilizes the generative capacity of diffusion models to represent multimodal value distributions. The core innovation of VDRL lies in the of the variational loss of diffusion-based value distribution, which is theoretically proven to be a tight lower bound for the optimization objective under the KI.-divergence measurement. Furthermore, introduce double value diffusion learning with sample selection to enhance training stability and further improve value estimation accuracy. Extensive experiments conducted the Mu- JoCo benchmark demonstrate that VDRL significantly outperforms some SOTA model-free online RL baselines, showcasing effectiveness and robustness.
Introduction
Recent advancements in model-free reinforcement learning (RL) integrated with diffusion models [I]
2,3,4 5 [7]
have demonstrated remarkable success in solving practical decision decision-making tasks, such
robotic locomotion control { robotic navigation (9] and robot manipulation [10] 11 [2].
By harnessing the expressive power of diffusion models 13 14 15), these works focus on enhancing the exploration efficiency of policy by effectively capturing the expressiveness and multimodality of policy beyond traditional action distribution assumption (e.g., unimodal Gaussian) [16]. However, a core challenge in RL remains the accurate estimation state-action values (i.e., Q-values), which is fundamental to both policy evaluation and improvement [17). Despite their enhanced exploration capabilities, RL methods combined with diffusion policies often suffer from value estimation errors, particularly overestimation [8] leading to suboptimal solutions [19].
To mitigate value overestimation.
techniques like double Q-network 20 and approximate upper-bound 21 have been incorporated into prominent RL. algorithms, such as twin delayed deep deterministic policy gradient (TD3) (22 and soft actor-critic (SAC) (23]. achieving significant perfor- mance improvements. Nevertheless, these methods may introduce a considerable underestimation bias and increase the computation cost for each iteration (24). Therefore, inaccuracies in value
estimation remain a limiting factor for further advancements in policy learning [25]
"Corresponding author <zhen.cuieee bnu.edu.cn, thang njust.edu.cn>.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

On the other hand, distributional RL 26), rooted in the idea of modeling the value distribution rather than estimating only the expected return (i.e.. Q-value), has emerged as a novel and promising avenue for improving the accuracy of value estimation. Building on this concept, some early 29
works [27 28
30 31 leverage quantile functions to represent the value distribution. However, these approaches are limited to discrete or low-dimensional action spaces. To address this challenge, the recent representative works such as DSAC [24] and DSACT [32] learn a continuous Gaussian value distribution and provide solid theoretical insights into how value distribution learning reduces overestimation. Despite the reduction in value overestimation, the parametric value distributions of these methods are constrained by the assumption (i.e., unimodal Gaussian), which may introduce bias between the approximated and true value distributions in tasks with non-stationary policies, where the true value distribution often exhibits multimodal characteristics. Consequently, this bias leads to imprecise value estimation, yielding a degraded performance. The above limitation motivates a key question: Can we develop universal value distribution representation of RI. that simultaneously achieves multimodal expressiveness and flexibility for complex continuous control tasks?
5 7 we propose Value Inspired by the considerable success of diffusion models applied in RL 4 Diffusion Reinforcement Learning (VDRL), a novel model-free online RL method. To the best of our knowledge, this is the first work to integrate diffusion models into distributional RL, enabling precise value estimation. The core idea of VDRL is conceptually elegant yet highly effective: we harness diffusion models to generate more expressive and multimodal distributions for representing value distributions. Our key theoretical insight reveals that the optimization objective of value distribution learning for distributional RL, under the KL-divergence measure, can be reformulated as the variational bound objective of the diffusion model. To mitigate variance introduced by iterative diffusion sampling, we introduce double value diffusion learning with adaptive sample selection, ensuring stable training while preserving distributional fidelity. In summary, the key contributions of this work are as follows:
i) To improve the accuracy of value estimation, we propose the value diffusion reinforcement learning (VDRL) method. VDRL leverages the capability of the diffusion model to fit more expressive and multimodal value distributions. More importantly, we theoretically prove that the variational loss of diffusion-based value distribution a tight bound for the optimization objective of value distribution learning under the KL-divergence measurement.
ii) To address the challenge of unstable learning introduced by iterative diffusion sampling, propose the double value diffusion learning with sample selection for value distributions. This mechanism can further reduce the value overestimation bias and might introduce minor value underestimation, which enhances learning stability and promotes better performance.
iii) We perform extensive experiments on eight popular MuJoCo 33 benchmark tasks, compar- ing the performance of VDRL to eleven model-free online RL baselines, including traditional RI, distributional RI., and diffusion- based policy RI methods. The experimental results demonstrate that VDRL achieves stable and consistently superior performance across most evaluated tasks, highlighting its robustness and effectiveness.
2 Preliminaries
2.1 Problem Formulation
The standard RL task could be regarded as a Markov decision process (MDP) 17
Formally, the task can be formulated as tuple M AS P,P, R, 7), where each element defined as follows:
S: finite set of states. Given a current state E S, we often denote the next state as s' E S.
A: set of actions. Note that the action may be discrete or continuous.
P(s'|s, a): the state transition function, which usually denotes the probability from the current state to the next state s' when the agent selects the action a.
R(s, a): the reward function. After the agent performs an action under the environment state s, the agent will receive an immediate global reward r R(s, a). Notably, in this work, the reward is explicitly treated as a random variable.
7 E (0, 1]: the discount factor used in the computation of cumulative return.

---

Further, the behavior of the agent is defined by stochastic policy π Ta]ss) which maps a given state probability distribution over actions. The state-action value function Q= a) associated with a given policy denotes the expectation of cumulative state-action return when the agent taking the action a E A under the state E S, i.e. Q (s, a) == F [ 7 R( 8k: 0&) [80 = 8, 8,00 = a].
2.2 Distributional Reinforcement Learning
Distributional Reinforcement Learning 26 29 learns the full probability distribution of the cumulative state-action return Z(-Is, a) instead of the expected return, i.e., single scalar Q-value Q a), which behaves better than traditional RL methods terms of learning stability and per- formance. In specific, the cumulative state-action return Z" a) associated with given policy ı is the sum of discounted rewards along the trajectory of interactions with the environment, i.c., 7 a) = 2k=0 > ag)) Note that the cumulative state-action return is a random variable, which describes the intrinsic randomness of the agent's interactions with its environment, i.e., the randomness of state transition P(s'|s, |s, a), reward R(s, a), and next state-action return ( a') (26). The distribution of the cumulative state-action return 2d (-1s, a) is a mapping from state-action pairs distribution over the random return, i.e., (s, a) 2d (1s,a) s,aa. Significant distributional RL is the use of the distributional Bellman operator to describe the value distribution, defined by
T 7(s, a) a) + 7 7(8' a'),
where the equation X .2 Y represents that the random variable X is distributed according to the same law with Y. s' ~ P(-|s, a). a' (-1ss), and Zs, a) is a new random variable, whose distribution is defined 7 (-)sa))
Distributional Soft Actor-Critic In most previous distributional RL methods, the value distribution is commonly modeled by discrete probability density function 26 30 29
quantile function [27 28 335 due to its highly expressive advantage [36]. However, these functions restrict direct access to the mean value, necessitating its approximation via weighted sum of finite quantiles. To overcome this issue, the recent work (24) proposed the distributional soft actor-critic (DSAC) method by directly learning continuous parameterized probability density function (Gaussian) for the random return (s, a). i.e., [=(+|s, a) = (QQoss a), o (s. a)). The parameter 0 can be updated by optimizing the Kullback-Leibler (KL) divergence between the target value distribution T 2 (-Iss, a) and value distribution (+|s, a) via gradient descent, i.e.,
(B) (s, [Dx Zz(+ss, a) || Zo(-|s, a))]
(2)
where B is replay buffer of previously sampled experience, and é represent the parameters of the target value distribution and policy functions.
2.3 Diffusion Model
Diffusion models 13 14] are highly powerful tools that learn to generate data by gradually denois-
ing normally distributed variable, which have achieved superior expressiveness in representing
complex probability distributions, particularly in image and video generation [37 38 39 40 41 A
representative algorithm is the well-known denoising diffusion probabilistic models (DDPM) (15). which synthesizes data by learning to reverse predefined noising process, i.e., forward process, through iterative denoising steps modeled by a neural network, i.e., reverse process. Given the real data Zo samples from the date distribution 2o q(zo), DDPM uses parameterized latent variable model, i.e., pe (2a) = f po )dE111 to model how the isotropic Gaussian noise FT ~ (O, I) is denoised into real data 2o, where z; denotes the step of diffusion process and T represents the total number of diffusion steps. The forward process is a Markov chain that gradually adds Gaussian noise to the data according variance schedule .,.... Br. expressed by
T
g(E1:r (o)) = II
g(x,(==11) = Ni /1 - 1,
(3)
t=1
The reverse process reconstructs data distribution starting at p(xT) A (xT; ; 0, I), represented as
(o(uu))) p(ær)
I
po (2)-1 (0( (( t), Eo (x),
(4)
[=1
For notational simplicity, this distribution is called as value distribution in this work.
3

---

where oi ( t) denotes the mean function approximator. and Ex(zt, t) is set I to untrained time dependent constants. Inspired by VAE 42 the variational lower bound (VLB) is utilized optimize the negative log likelihood as the training objective of DDPM:
Po (Zo:T)
(0) Exoo Poo--oo-
log ( (mn)
(5)
To this end, the diffusion model is trained by minimizing the simplified loss function:
LDEPM (0) [1.])..]..0.. ||c colva + vi- o
(6)
where O2
II Us 04 = 1 - Br and is sampled from from.N (0,
3 Method
In this section, we propose novel model-free online RL method called value diffusion reinforcement learning (VDRL) to improve the accuracy of value estimation. The key innovation of VDRL is the use of the variational loss of diffusion-based value distribution, which can be theoretically proven as a tight lower bound for the optimization objective of value distribution learning under the KL- divergence measurement. Furthermore, we introduce double value diffusion learning with sample selection to stabilize training and further enhance estimation accuracy.
3.1 Variational Objective for Value Difftusion Learning
According the distributional Bellman operator in Eq. (T. the optimization objective of value distribution learning in distributional RL is to minimize the distance between the target value distribution, Zur (+ss, a), and the current value distribution, ier (-Is, a). formally.
min Zoe (e.0)~B
E.
[d ()), Zeur (18,
where (,,(()),, Zou (-18, a) and d is the distance metric for measuring two probability distributions, such as Kullback-Leibler (KL) divergence, Wasserstein distance.
To optimize the objective in Eq. 7 practice. we need choose a tractable approximating distribution for modeling the value distribution. For instance, the value distribution is modeled as unimodal Gaussian distribution in DSAC (24 and DSACT [32.. However, as highlighted in [3), the unimodal Gaussian approximation fails to capture the inherent expressiveness and multimodality of value distributions, which may introduce bias between the approximated and true distributions in tasks with non- statsoationar policies, leading to imprecise value estimation and then yielding degraded performance. More importantly, we find that when the value distribution modeled by a diffusion model, the optimization objective under the KL-divergence measurement can be reformulated and simplified into a variational bound objective, which formalized follows:
Theorem 1. (Lower Bound of Learning Objective for Value Distribution) If the value distribution is modeled by a diffusiin model, the variational bound objective
Po Po Zo.T
((s.ans')-B.a' BT= 2111aam00022.....7727 log 2(T** a T Zo) 2(d a a'))=
(8)
the tight lower bound of the optimization objective of value distribution learning for distributional RL under the KL-divergence measurement
E(g.ay- [T 2p(-1s, ()())s,)))
(9)
where Zo (o(-)8,0) a) denotes the value distribution to be optimized and Ez{-18,0) a) is the parameterized target value distribution, and the equality holds when the policy converges.
Proof. The proof is deferred to Appendixx A
Remark 1. Although the variational lower bound objective in our method (VDRL.), as shown in Eq. (8 appears similar to QVPO (6), they differ fundamentally in both derivation and formulation:

---

Derivation: Our variational lower bound is derived from the optimization objective of the value distribution, whereas QVPO derives its lower bound from the policy objective in RL.
Formulation:
The variational lower bound objective QVPO incorporates Q-value under the specific condition, i.e., Q(s, a) 2 0, while our method maintains a straightforward form without introducing additional constraints.
According to Theorem | we can derive the variational loss function of value diffusion learning as Eq. (1O), which be applied to value distribution optimization for distributional RL.
(())= = E Z(s'
||ee EB (VārT" vI âGe, s,a,), 11.]
(10)
Where 7 as) R(s,a) ( a') (lW
3.2 Double Value Diffusion Learning with Sample Selection
While the value diffusion learning captures the inherent expressiveness and multimodality of the true value distribution, it also introduces large variance, which may lead to unstable learning processes in some tasks. To address this problem, we propose double value diffusion learning with sample selection for value distribution 2(-Is, a). Specifically, we parameterize two value distribution functions Zo (-|8,
[ss a). then choose sample with lower random return Z(s', a') to calculate the target random return T (s, a), formally,
Z*, R(s,a) + min Z(s', (z)t (1 (1s',a))-
(11)
im1
Furthermore, as shown in Table this mechanism for value distribution can further reduce the value overestimation, and might induce relatively minor value underestimation. In fact, as discussed in TD3 [22], the slight value underestimation preferable value overestimation, as unlike overesti- mation, underestimated value do not propagate explicitly through policy update during training.
3.3 Value Diffusion Reinforcement Learning
According the above introduction, present practical model-free online off off-policy RL algo- rithm called value diffusion reinforcement learning (VDRL), which includes two value distribution functions ((||s,a) a) and policy function [a]ss). To that end, both the value distribution and the policy, and alternate hetween optimizing both networks with stochastic gradient descent. Specifically,,
function approximators for
consider two parameterized value distribution func- tions Ze, (-Is, a), (+|s, a) and a parameterized policy function (als). The parameters of these networks are 0,, 02. and Φ, respectively. Furthermore, we employ the idea of a target network, where parameters of the target network can be an exponentially moving average of the main network weights, which has been shown stabilize training DQN 44. According the theorem [ in section 3.1 the value distribution functions can be trained to minimize the variational objective,
(T Zo.r 1s, a) log "(T "1. 8 a, T Zo)
Jz
E. (a.a.n.')) ). B' wag Z(e' 2')-22,,aaa0 (1l''aa) a²)
(rr a. T Ta)
(12)
Then, the policy function parameters can be learned by directly maximizing the expected cumulative state-action return (s,, a), formally,
J (Ø) Eg min Bz('', (')) (Z(s.a))
(13)
In implementation, we employ Monte Carlo estimation to approximate the expectation of each value distributions, i.e., Ez(s (')=z,"))ZZ, (s, (s',a") [Z(s, a)). For the sake of comprehensiveness, we provide a complete training process of the proposed VDRL method, shown in Algorithm |
4 Related Work
In this section, we will look over existing works that are the most relevant to our proposed method. We divide our surveys into two main parts. Firstly, we provide overview of distributional reinforcement
5

---

Algorithm 1 Value Diffusion Reinforcement Learning
Input: Diffusion- value distribution networks { {Zo, (Z]s, a)}² target diffusion- value distribution networks (2g, (2|s, d)} policy network (a])s,), target policy network Tg (als), the learning rate of value distribution and policy ira, Ir the momentum coefficient
1: Initializing: all networks parameters 01 01, B2 t 02 2
2: for each iteration do
3: Initialize the environment;
4: for each sampling step do
5: Select action
(T a);
6: Execute action a to obtain the reward r and the next state s';
7: Store (s, a, r, done) in replay buffer B;
8:
end for
9:
for each optimization step do
10:
Sample random minibatch of N samples from replay buffer: a, r'' ddone;;
Update the parameters of diffusion baased value distribution networks {0:}7-i 0, t 0B Bg Va, (Z,);
12:
Update the parameters of policy network di d + & Φ 3, (z)())
13:
Update the parameters of target diffusion- basion value distribution networks 0, = T0, + (1
r)0 for ė = {1,2}:
14:
Update the parameters target policy network ē = Tō + (1 T)Φ;
15:
end for
16:
end for
learning. Secondly, we discuss the diffusion-based RL methods that use diffusion models as polices for model-free online RL setups.
Distributional Reinforcement Learning. Recent interest in distributional methods 45 46 for RL has grown with the introduction of deep RL approaches for learning the value distribution. This paradigm was first formalized the seminal work [26]. which introduced the Categorical DQN (C51) algorithm that approximates the value distribution using fixed categorical distribution over discrete support points and applies a distributional Bellman operator. Building on this foundation,
subsequent works [27 28 29
30 31 leveraged quantile functions to represent the value distribution, enabling significant advancements. However, these approaches are inherently limited to discrete and low-dimeniional action spaces. In order deal with continuous control settings, methods such
as 34 35 47 (88 have been proposed in recent years. Nevertheless, these approaches approximate the mean value through a weighted average of finite quantiles, which may reduce the accuracy of the mean value estimation, leading to suboptimal performance in some tasks. The distributional soft actor-critic (DSAC) 24 addressed the above limitation by learning continuous Gaussian value distribution with the Q-value directly parameterized by the critic network, improving value estimation accuracy. Despite its effectiveness, DSAC suffers from training instability and sensitivity. To alleviate these issues, the term DSAC with three refinements (DSACT) (32) introduces three key refinements to refine Q-value estimation. However, both DSAC and DSACT constrain the value distribution and stochastic policy space to unimodal Gaussian distributions, which hinders exploration efficiency and learning performance, particularly in tasks with multiple goals or complex dynamics, where the true distribution often exhibits multimodality. In contrast, the proposed VDRL method employs diffusion models to capture more expressive and multimodal representations of the value distribution, which further enhances value estimation and learning performance in tasks requiring sophisticated modeling of uncertainty and dynamics.
Diffusion Models for Model-free Online Reinforcement Learning, Recently, numerous works [I] 53 54 2 3 have employed diffusion models the policy class in model-free offline tasks, due to their superior expressive richness in representing complex contrast, diffusion policies have been less widely explored RL, owing to the noisier and policy-varying nature of value estimations [16]. To date, only several have investigated the application of diffusion models for online RL. The pioneering work, DIPO provided a solid convergence guarantee for diffusion policies and facilitated policy improvement by updating the action via the action gradient of the Q-value. However, fully learning of action gradient not only presents significant challenge but also introduces additional computational
49 50, 5T
52
RL to tackle www.sion-maaeeeee distributions.
model-free online
studies 4 5 6.

---

costs. Unlike DIPO, Q-score matching (QSM) 5 proposed a novel policy update mechanism by aligning the score of the learned diffusion model, i.e., V Volog(<(a(s)) with the action gradient of the Q-value. Unfortunately, it neglects inaccuracies in the value function gradient and biases introduced during the alignment process when updating the policy. To mitigate these limitations, Q-weighted variational policy optimization (QVPO) [6] introduced the Q-weighted variational lower bound (VLO) loss, and then designed a tailored entropy regularization term for diffusion polices. Concurrently, Wang et al. 77 proposed the diffusion actor-critic with entropy regulator (DACER) method, and employed Gaussian mixture models for entropy estimation. Despite these great advances, fundamental challenges persist: approximation inaccuracies and computational inefficiencies continue to constrain the effectiveness of diffusion models in model-free online RL. Different from all existing from model-free free online RL methods with diffusion policy, from the value perspective, the proposed VDRL method utilizes diffusion models to represent value distributions, which enhances the accuracy of value estimation, yielding significantly improved performance.
5 Experiments
In this section, we empirically validate the effectiveness of the proposed algorithm on different decision-making tasks and with various hyperparameters. With these experimental results, we aim to answer the following questions: 1) How does VDRL compare to traditionally canonical and existing diffusion-basse model-free online RL methods? 2) What contributes the performance improvement of VDRL on the continuous locomotion tasks? Note that all experiments are conducted on 2.90GHz Intel Core 17-10700 CPU, 64G RAM, and NVIDIA GeForce RTX 3090 GPU.
5.1 Experimental Setup
Evaluation Environment. In this work, we use eight different level tasks of the popular MuJoCo B3] benchmark
P to evaluate the performance of all methods, including Ant-v3, HalfCheetah-v3, Hopper- v3, Humanoid-v3, Inverted2Pendulum--e Pusher-v2, Swimmer-v3, and Walker2d-v3. The more details of these tasks and the MuJoCo benchmark are deferred to AApeeddxx B
Compared Baselines. We evaluate the performance of VDRL against eleven model-free online RL algorithms, categorized into three groups. The first group comprises traditional model-free RL baselines, including two on-policy methods (PPO (55) and SPO (56)) and three off- off-policy algorithms (DDPG (57). TD3 22). and SAC (23]. The second group focuses on distributional RL approaches, represented by DSAC (24) and DSACT [32]. Finally, the third group consists of advanced diffusion- based policy RL methods, including DIPO 4. QSM [5). QVPO (6). and DACER [7]. The more details of implementation and hyperparameters for all baselines are deferred to Appendix D.
Methodology for Reported Metries. For the proposed method and all baselines, all experiments are conducted over 800,000 training interaction steps with four runs and different random seeds (0, 50, 100, 200), with each performing one evaluation rollout every 10,000 interaction steps. In particular, each evaluation result is the average of ten episodes. The mean return, standard deviation, and average highest return are logged as the performance of methods.
5.2 Details of Implementation
The value distribution is modeled using the reverse process of conditional diffusion model, parame- terized similarly to DDPM [15], as detailed in Section 2.3 The effects of reverse diffusion steps and noise schedules are analyzed in Section [5.4] For the policy, Gaussian distribution with mean and diagonal covariance is adopted. The policy network architecture consists of two hidden layers, each with 256 units and GeLU activation functions. Parameter updates are performed using the Adam optimizer. A detailed description of all training hyperparameters is deferred to Appendix C
5.3 Comparative Evaluation
The best performance of the proposed method (VDRL) and all baselines across eight MuJoCo benchmark tasks is presented in Table [ VDRL achieves results comparable to baseline methods
'The version of MuJoCo utilized in all experiments of this work is mujoco210, which is the same version used as DSAC 24 QVPO 66. DACER 77. and DSACT 32
7

---

Table I: Performance on eight tasks of OpenAl Gym MuJoCo benchmark. The results show the best
mean returns and standard deviations over 800,000 training interaction steps and four random seeds.
ANT-V3
HAL#CHERTAH-V2
HOPPER-V
HUMANOOD-v3
PPO
2129.33
1757.57 25.86
3291.76
612.15
Traditional
SPO
56
2019.02 = 46.91
4725.37
102.64
3480.20
92.66
749.70 = 131.01
DDPG
57
1124.35 1 51.49
8983.40
140.39
3326,
12.22
597 52 1: 104.79
Model-Free
1D3
2836.68 = 11.57
8306.89 = 82.31
3197.59 = 1.39
50000.91 = 13.38
23
12700.1 130.00
10214.06
3380).51
43.05
DSAC
24
1261195 215.50
11610.33 =
3436 95 13.11
5009 60 271.91
Distributional RI.
DSACT k2
5313.04 + 122.46
11670.82 + 131.81
3654. 30 = 7.06
5087.90 0087.93-00 165.00
DIPO 4
6224.93 H: 82.27
10062.71 49.81
3454.60
37.90
5266.87 +: 5.93
Diffusion-basee
QSM 3
839.53 + 510.14
9204.16 + 85.61
3513.23
22.65
1950 42 + 1270.96
Policy
QVPO 0
5222.62 78.02
90667
72:47
3369.51
5252.72 13.75
DACER
5506.40 +: 99.67
11350.96
119.48
3388.54
2.79
4447.02 750.42
VDRL (Ours)
7236.67 1. 89.44
11815.20
142 04
3679.41
7.47
5497.15 48.58
METIOD
PUSHER--V2
SWIMMER-V3
WALKER2D-V3
INVERTEO22PNDDULUU--W
PPO 55
-22.92 +: 3.47
137.59 = 1.35
3168.82 + 101.73
9359.67 1: 01221
Traditional
SPO
22.29
3849.82 245.47
9359.33 0.35
Model-Free
DDPG 57
335.3 1. 4.94
42.64 1.70
3478.72 1185.37
9359.74 0.22
1D3
-23.06 2.33
73.78 :: 0.88
383-1 1: 64.33
13 = 17 69
SAC
20.28 1: 1.62
77.49
3.07
4486.88 42.63
9359 83 1. 0.15
DSAC
24
-31.43 3.93
140.76 + 9.20
4819.56 + 171.21
9359.83 0.08
Distributional RL
DSACT
-25.65 2.63
134.46
5134.79 79 22.68
9359. 75 0.11
DIPO 9
39.60 + 5.50
55. 10 2.01
4508 (088.09-22 10.41
9359.85 0.16
Diffusion-basee based
QSM 5
46.69 1. 3.41
105.04 1. 2.77
3897.01 1 721.01
2475.43 1. 89.95
Policy RL.
QVPO 6
38.52 + 5.74
124.14 + 1.44
4198.86 46.44
9354.38 0.86
224.91 + 3.23
151.55 + 1.04
4053. 14 62.
9359.79 0.14
VDRL (Ours)
-26.92 3.95
144.40 +: 2.27
77 10.30
9359. 0.16
on simpler tasks such as Pusher-v2, Swimmer-v3, and InvertedPendulum--2. while significantly outperforming them on more complex locomotion tasks, including Ant-v3, Hopper-v3, Humanoid-v3, and Walker2d-v3. The Figure [ Iillustrates the learning curves of all algorithms, further demonstrating that VDRL delivers stable and consistently high performance across all evaluated tasks, highlighting its superior robustness. In contrast, baseline methods struggle with poor performance on one or more tasks, emphasizing the limitations of existing approaches.
(a) Ant-v3
(b) HalfChectah-v3
(c) Hopper-v3
(d) Humanoid-v3
(c) Pusher-v2
Swimmer-y3
Walker2d-v3
(h) Inverted2Pendulum-22
Figure 1: Learning curves of the proposed method and all baselines 8 continuous tasks of MuloCo benchmark over 10 evaluation episodes every 10k iterations, where the solid lines correspond to the mean and the shaded regions standard deviations over different random seeds.
Table 2 shows the average value estimation bias, i.e., the difference between the estimated Q-value and the true Q-value, for VDRL and some baselines, including traditional model-free online RI. methods and distributional RI. approaches. The true Q-value approximated by the average actual discounted cumulative return across states over ten episodes, evaluated every 10,000 iterations. Notably, the
8

---

value estimation bias for the InvertedPendulum-v22 task is excluded, effective policies are learned before the value function converges. The experimental results yield several key insights. First, compared to conventional RI. algorithms, both VIRLL and distributional RI. baselines demonstrate reduced value estimation bias in most tasks, underscoring the effectiveness of value distribution learning in mitigating estimation inaccuracies. Second, VDRL consistently achieves lower bias than DASC and DSACT, highlighting the advantages of leveraging diffusion models to capture multimodal and complex value distributions for improved estimation accuracy. Last, even when compared to on-policy RL methods such as PPO and SPO, VDRL exhibits a significant advantage in value estimation accuracy across most tasks, further validating its effectiveness.
Table 2: Average value estimation bias over four runs for the proposed VDRL method, conventional model-free RL, and distributional RL baselines on eight tasks of MuJoCo benchmark. In particular, the bias is calculated as the difference between the estimated Q-value Qest and the true Q-value Qiee. i.e., Qest mm. where the true Q-value is accessed based on the discounted accumulation of rewards. Furthermore, the name of each task are simplified because of the space limitation.
METHOD
ANT3
HAL3
HoP3
HUM3
Pus2
Sw13
WAL3
Traditional on-policy RI.
PPO
9.45
320.36
271.29
16.74
18.05
0.94
2.51
SPO
56
11.52
182.19
275.74
15.26
21.35
0.88
3.60
DDPG 57
94.63
37.27
476.89
49.08
7.93
12.30
126.04
Conventional off-policy RL
TD3
373.20
372.05
929.33
225.06
19.76
-3.07
-88.56
SAC
23
27.94
-6.28
251.92
-92.48
- 13351
-1.06
-3.14
DSAC
37.94
83.47
2264.98
65.72
5.15
1.84
69.93
Distributional RL
DSACT
25.17
36.76
-181.54
59.24
4.78
-1.13
-9.00
VDRL (Ours)
-7.36
4.13
113.70
-12.69
6.85
0.42
0.95
5.4 Sensitivity Analysis
We perform sensitivity analysis on different diffusion steps and diffusion noise schedule, using the Ant-v3 and Humanoid-v3 tasks as examples.
Diffusion Steps. We evaluate the performance of VDRL under varying diffusion steps (7 = 5, 10, 20, 30), shown in Figure 2(a) 2(b) The results indicate that the performance does not improve monotonically with an increasing number of diffusion steps. Adopting larger diffusion steps degrades the performance of VDRL, potentially due to gradient vanishing or exploding issues. Additionally, larger diffusion steps will lead to increased training and evaluation costs. To balance the performance and computational efficiency, we choose 10 diffusion steps for all tasks.
Diffusion Noise Schedule. Figure 2(d) illustrates the performance of VDRL with distinct diffusion noise schedules. It can be observed that the cosine and variance preserve schedule obtain comparable results, and both outperform the linear schedule. Thus, we select the cosine schedule for all experiments.
(a) Diffusion Steps
(b) Diffusion Steps
(c) Noise Schedule
(d) Noise Schedule
Figure 2: Ablation learning curves for different diffusion steps and diffusion noise schedule on Ant-v3 and Humanoid-v3 tasks, where the solid lines correspond to the mean and the shaded regions to standard deviations over 4 different random seeds.

---

Conclusion and Discussion
In this work, we presented a novel model-free online RL method called value diffusion reinforcement learning (VDRL) overcome the representational limitations of learned value distributions. By harnessing the inverse denoising process of diffusion models, VDRL captures expressive, multimodal value distributions, thereby substantially improving value estimation. The theoretical cornerstone of VDRL lies in its diffusion-based variational loss, which we rigorously establish as a tight variational lower bound for optimizing value distributions under the KI.-divergence measurement. Furthermore, to improve the training stability and further enhance value estimation, we introduce double value diffusion learning with sample selection. Comprehensive experiments on eight continuous control tasks from the MuJoCo benchmark demonstrate the effectiveness and robustness of VDRL.
Limitations. While value diffusion learning presents promising advance, its application to online RL faces several unresolved challenges. For instance, the reliance on iterative value diffusion sampling incurs significantly increased computational costs than traditional model-free RL and distributional RL methods, especially in high-dimensional action spaces. In addition, this work is evaluated primarily on MuJoCo benchmarks, leaving its scalability real-world robotics applications with partial observability and sensor noise as an open question.
In our future research, will pursue two key avenues to advance this paradigm. First, we will develop lightweight diffusion models, potentially through knowledge distillation, to reduce the training and inference latency. Second, we will explore hybrid methods that integrate value diffusion with model-base -ased planning, aiming to tackle the challenges of complex, long- monghorizon tasks and thereby unlock further performance gains in online RL.
Future Work.
Acknowledgement
This work was supported by the National Natural Science Foundation of China (Grants No. 62476133) and the Fundamental Research Funds for the Central Universities (Grant No. 11300-312200502507)
References
[1] Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, pages 9902-9915 PMLR, 2022.
[2] Bogdan Mazoure, Walter Talbott, Miguel Angel Bautista, Devon Hjelm, Alexander Toshev, and Josh Susskind. Value function estimation using conditional diffusion models for control. arXiv preprint arXiv:2306.072000 2023.
[3] Zhengbang Zhu, Minghuan Liu, Liyuan Mao, Bingyi Kang, Minkai Xu, Yong Yu, Stefano Ermon, and Weinan Zhang. Madiff: Offline multi-agent learning with diffusion models. In Advances in Neural Information Processing Systems, volume 37, pages 4177-4206, 2024.
[4] Long Yang, Zhixiong Huang, Fenghao Lei, Yucun Zhong, Yiming Yang, Cong Fang, Shiting Wen, Binbin Zhou, and Zhouchen Lin. Policy representation via diffusion probability model for reinforcement learning. arXiv preprint arXiv:2305. 13122, 2023.
[5] Michael Psenka, Alejandro Escontrela, Pieter Abbeel, and Yi Ma. Learning diffusion model policy from rewards via q-score matching. International Conference on Machine Learning, pages 41 41163-411 1882. PMLR, 2024.
[6] Shutong Ding, Ke Hu, Zhenhao Zhang, Kan Ren, Weinan Zhang, Jingyi Yu, Jingya Wang, and Ye Shi. Diffusion-based reinforcement learning via q-weighted variational policy optimization. In Advances in Neural Information Processing Systems, volume 37, pages 54183-54204, 2024.
[7] Yinuo Wang, Likun Wang, Yuxuan Jiang, Wenjun Zou, Tong Liu, Xujie Song, Wenxuan Wang, Liming Xiao, Jiang Wu, Jingliang Duan, et al. Diffusion actor-critic with entropy regulator. In Advances in Neural Information Processing Systems, volume 37, pages 54183-54204, 2024.
10

---

[8] Xiaoyu Huang, Yufeng Chi, Ruofeng Wang. Zhongyu Li, Xue Bin Peng, Sophia Shao, Borivoje Nikolic, and Koushil Sreenath. Diffuseloco: Real-time legged locomotion control with diffusion from offline datasets. arXiv preprint arXiv:2404. 19264, 2024.
[9] Gengyu Zhang, Hao Tang, and Yan Yan. Versatile navigation under partial observability via value-guided diffusion policy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17943-179511 2024.
[10] Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, page 02783649241 273668, 2023.
[11] Tsung- Wei Ke, Nikolaos Gkanatsios, and Katerina Fragkiadaki. 3d diffuser actor: Policy diffusion with 3d scene representations. arXiv preprint arXiv:2402.10885, 2024.
[12] Paul Maria Scheikl, Nicolas Schreiber, Christoph Haas, Niklas Freymuth, Gerhard Neumann, Rudolf Lioutikov, and Franziska Mathis-Ullriche Movement primitive diffusion: Learning gentle robotic manipulation of deformable objects. IEEE Robotics and Automation Letlers, 2024.
[13] Jascha Sohl-Dickstein, Eric A Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, volume 37, pages 2256-2265. PMLR, 2015.
[14] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, volume 32, pages 11918-119300 2019.
[15] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Pro- ceedings of the 34rd International Conference on Neural Information Processing Systems, 33:6840-6857 2020.
[16] Zhengbang Zhu, Hanye Zhao, Haoran He, Yichao Zhong, Shenyu Zhang, Haoquan Guo, Tingting Chen, and Weinan Zhang. Diffusion models for reinforcement learning: survey, 2024.
[17] Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction. MIT press Cambridge, 1998.
[18] Hado Hasselt. Double q-learning. In Proceedings of the 24th International Conference on Neural Information Processing Systems-Volue Volume 2, pages 2613-2621 2010.
[19] Sebastian Thrun and Anton Schwartz. Issues in using function approximation for reinforcement learning. In Proceedings of the 1993 connectionist models summer school, pages 255-263, 1993.
[20] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence-Volie Volume 30, pages 2000- 2016.
[21] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018
[22] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.
[23] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference machine learning, pages 1861-1870. PMLR, 2018
[24] Jingliang Duan, Yang Guan, Shengbo Eben Li, Yangang Ren, Qi Sun, and Bo Cheng. Distribu- tional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors. IEEE transactions neural networks and learning systems, 33(11):6584 (5544)6684-658 2021.
11

---

[25] Xiangkun He, Wenhui Huang, and Chen Lv. Toward trustworthy decision makiing for au- tonomous vehicles: A robust reinforcement learning approach with safety guarantees. Engi- neering, 33:77-89, 2024.
[26] Mare G Bellemare, Will Dabney, and Rémi Munos. distributional perspective on reinforce- ment learning. In International conference on machine learning, pages 449-458. PMLR, 2017.
[27] Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In International conference machine learning, pages 1096-1105. PMLR, 2018.
[28] Will Dabney, Mark Rowland, Mare Bellemare, and Rémi Munos. Distributional reinforce- ment learning with quantile regression. In Proceedings of the AAAI conference on artificial intelligence-Volle Volume 32. pages 2892-2901, 2018.
[29] Borislav Mavrin, Hengshuai Yao. Linglong Kong, Kaiwen Wu, and Yaoliang Yu. Distributional reinforcement learning for efficient exploration. In International conference on machine learning, pages 4424 PMLR, 2019.
[30] Mark Rowland, Robert Dadashi, Saurabh Kumar, Rémi Munos, Mare G Bellemare, and Will Dabney. Statistics and samples in distributional reinforcement learning. In International Conference on Machine Learning. pages 5528-5536 PMLR, 2019.
[31] Daniel Brown, Scott Niekum, and Marek Petrik. Bayesian robust optimization for imitation learning. In Advances in Neural Information Processing Systems, volume 33, pages 2479 2491, 2020.
[32] Jingliang Duan, Wenxuan Wang, Liming Xiao, Jiaxin Gao, Shengbo Eben Li, Chang Liu, Ya- Qin Zhang, Bo Cheng, and Keqiang Li. Distributional soft actor-critic with three refinements. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(5):3935-3946, 2025.
[33] Emanuel Todorov, Tom Erez, and Yuval Tassa Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012.
[34] Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, TB Dhruva, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. In International Conference on Learning Representations, 2018.
[35] Chen Tessler, Guy Tennenholtz, and Shie Mannor. Distributional policy optimization: an alternative approach for continuous control. In Advances in Neural Information Processing Systems, volume 32, pages 1352-1362, 2019.
[36] Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International conference machine learning, pages 1747-1756. PMLR, 2016.
[37] Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. In Advances in Neural Information Processing Systems, pages 8780-8794, 2021.
[38] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127 2023.
[39] William Peebles and Saining Xie. Scalable diffusion models with transformers. In IEEE/CVF international conference on computer vision, pages 4195-4205. 2023.
[40] Yuanzhi Wang, Yong Li, and Zhen Cui. Incomplete multimodalityddiffuse emotion recognition. Advances in Neural Information Processing Systems, 36:17117-17125 2023.
[41] Yuanzhi Wang, Yong Li, Mengyi Liu, Xiaoya Zhang, Xin Liu, Zhen Cui, and Antoni B Chan. Re-attentional controllable video diffusion editing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 1131,
12

---

[42] Diederik P Kingma and Max Welling. Auto-eng variational bayes, 2022.
[43] Liming Xiao, Yao Lyu, Fawang Zhang, Liangfa Chen, Guangyuan Yu, Shengbo Eben Li, Fei Ma, and Jingliang Duan. Multi-style distributional soft actor-critic: Learning a unified policy for diverse control behaviors. IEEE Transactions on Intelligent Vehicles, pages 1-12, 2024.
[44] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518((540))5599555 -33,, 2015.
[45] Will Dabney, Zeb Kurth-Nelson, Naoshige Uchida, Clara Kwon Starkweather, Demis Hassabis, Rémi Munos, and Matthew Botvinick. A distributional code for value in dopamine-based reinforcement learning. Nature, 577(7792):671-673 2020.
[46] Mare G Bellemare, Will Dabney, and Mark Rowland. Distributional reinforcement learning. MIT Press, 2023.
[47] Yangang Ren, Jingliang Duan, Shengbo Eben Li, Yang Guan, and Qi Sun. Improving general- ization reinforcement learning with minimax distributional soft actor-critic. In 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), pages 1-6. IEEE, 2020.
[48] Yash Chandak, Scott Niekum, Bruno da Silva, Erik Learned-Miller, Emma Brunskill, and Philip S Thomas. Universal off-policy evaluation. In Advances in Neural Information Processing Systems, volume 34, pages 27475-27490, 2021.
[49] Wenhao Li, Xiangfeng Wang, Bo Jin, and Hongyuan Zha. Hierarchical diffusion for offline decision making. In International Conference on Machine Learning, volume 202, pages 20035-20064. PMLR, 2023.
[50] Fei Ni, Jianye Hao, Yao Mu, Yifu Yuan, Yan Zheng, Bin Wang, and Zhixuan Liang, Metadiffuser: Diffussonn model as conditional planner for offline meta-rl. In International Conference on Machine Learning, pages 26087-26105. PMLR, 2023.
[51] Zhixuan Liang. Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, and Ping Luo. Adaptd- iffuser: Diffusion models as adaptive self-evolving planners. In International Conference on Machine Learning, pages 20725-20745. PMLR, 2023.
[52] Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou, Diffusion policies an expressive policy class for offline reinforcement learning. In International Conference on Learning Representations, 2023.
[53] Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.1053 2023.
[54] Huayu Chen, Cheng Lu, Chengyang Ying. Hang Su, and Jun Zhu. Offlie reinforcement learning via high-fidelity generative behavior modeling. In International Conference Learning Representations, 2023.
[55] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint urXiv:1707.0634/ 2017.
[56] Zhengpeng Xie, Qiang Zhang, Fan Yang, Marco Hutter, and Renjing Xu. Simple policy optimization. arXiv preprint arXiv:2401.16022 2025.
[57] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning, pages 387-395. PMLR, 2014.
[58] Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https: :// thub. com/ opt/aaa/baallleees 2017.
13

---

A Proof
Theorem 1. (Lower Bound of Learning Objective for Value Distribution) If the value distribution is modeled by diffusion model, the variational bound objective
E ) B 2ur 2umla,,,7 log
Z0: 8. a) q(7 1:2 (1 a, T Zu
]1:
(14)
Z(o' a''' 2'))-z((166'aa))
the tight lower bound of the optimization objective of value distribution learning for distributional RL under the KL-divergence measurement
B( (T 2g(11s,) 20(-)s,a))],
(15)
where Zo(1 B-(-ss,0) a) denotes the value distribution to be optimized and Z8(-|s, 0) is the parameterized target value distribution, and the equality holds when the policy converges.
Proof. The optimization objective of value distribution learning under the KL-divergence measure- ment is
min E (T" 2z(1|s, Zo(-|s, (a.a)~B
(16)
where B is the replay buffer of historical samples, 20(-|s, a)) denotes the value distribution to be optimized, and Z a)) is the parameterized target value distribution. According the definition of KL-divergence, we can show that
(z) = E (H,a)-
[KL (7(+) || Zo(-ss,))) a))]
IT ((s,a)
((, ((TT
E (a.a~B
Pa
E
(s. a))
tc
(17)
(s.0)~B T=2(s,a)
E
[ET (77221.0)77 llog
+c
(8,0)~B
(=..rr")-..ddw )
where the constant e denotes the term independent of 0, मद denotes the parameterized target policy, and T (s, a) = R(s, a) + (s', a'). For simplicity of exposition, we let Y denotes T (s, a). Thus, the optimization objective of value distribution learning can be formalized as follows:
max
[o)P
(18)
B.a. wx Z(e' Z'aaaa'"
(,)))),)) is diffusion-based value distribution, we have
Po(Yi.T(s, Yo)
(Y) -Exir (Viraa(())
log Po(Y a, Yo) Pa
Po(Yo.Tlo, Yo)
log -Vy.aro(Vir
Po(Yi-rss,a,1)
log Po s, a) ((irlss,,)))
-By
q(Yirrls, Yo) Po
PY(ooTlss,))
log =Byirsg
+ (oa Yo) Po(Y:rls,, Yo)).
(Virr1 Yo)
(19)
14

---

According to the inequality of Jensen, the term a, Yo) Po a, Yo)) in Eq. 19 can be shown that
q( Yo)
- r D Yo) a,
log
Po(Y Yo)
(
By 1yy
log q( g(Vi.T 8, Yo)
Po 8, Yo)
- log (Milsa(())
g(Yi,T Yo)
(20)
Po Po is equal to 1, and show that ((1,r1s;,;
When the learning policy converges, the term
Jz(0) = E
DKL 2z(1s,
20(-)s,)))
(a.0)~18
E
logPo(Y))
(21)
Po Po(Yorlss,r s, (YiTT Yo)
E B.a. Z(d' a'd 2-1--'''''
E Ey
log
Thus, the variational bound objective
E Z log Po 8,0) a)
E (Bs.nn)) ((4
(22)
g T= Zo)
the tight lower bound of the learning objective of value distribution for distributional RL under the KI-ddiergeecce measurement
(s.a)= [Ki 2g(+Is,) (T"(((()))))) || 2 ())]..)
(23)
This completes the proof of Theorem I
B Environmental Details
MuJoCo (Multi-Joint dynamics with Contact) 33 is high-performance physics engine widely used roboties and biomechanics simulations, renowned for efficiency and accuracy in modeling complex dynamics. It provides versatile platform for developing and benchmarking reinforcement learning (RL) algorithms, enabling the simulation of articulated bodies with intricate interactions, including friction, contact, and soft constraints. These features make it particularly suitable for tasks involving multi-joint movement and physical interactions.
In this work, we use eight MuJoCo benchmark tasks, categorized into three classes based on their dynamics and objectives: locomotion, control, and navigation. Specifically, the locomotion tasks focus on the ability of agents move efficiently across terrain, including Ant-v3, HalfCheetah-vä Hopper-v3, Humanoid-v3, and Walker2d-v3. The control tasks, such as Inverted2Pendllum-y and Pusher-v2, require agents to manipulate objects maintain certain positions. The navigation task, exemplified by Swimmer-v3, evaluates the agent's 's ability to move through a fluid medium. Figure shows the visualization of all tasks used in this study. Furthermore, these tasks are detailed follows:
Ant-y3: (s x a) E IIll x R a four-legged robot learns to navigate and cover distance by coordinating its legs, with key challenges in balancing and adapting to varying terrains.
HalfCheetah-v3: (s x a) E R!7 x R a bipedal robot emulates the running motion of cheetah, with the objective maximize forward speed while maintaining stability.
Hopper-v3: a) E IR1 x R33 single-legged robot learns to hop forward efficiently, focusing on balance and joint control to maximize distance.
Humanoid-v3: (s x a) 3776 R47,, humanoid robot must learn to walk and run, tackling the complex challenge of maintaining balance during motion.
15

---

(a) Ant-v3
(b) HallCheetah-v3
(c) Hopper-vv
(d) Humanoid-v3
(e) Pusher-v2
Swimmer-v3
Walker2d-v3
(h) Inverted2Pendllum-22
Figure 3: The screenshots of MuJoCo tasks utilized in this work.
Walker2d-v3: (s x a) R7 x R a two-legged robot learns to walk forward, emphasizing coordination and adaptability in dynamic environments.
(s a) E R x R classic control task where the agent balances an inverted pendulum on a pivot, requiring precise application forces to maintain stability. Pusher-v2: (s a) 223 R², the agent controls robot arm push a block to target location, testing precision in force application and obstacle navigation.
Inverted2Pendullu-
Swimmer-v3: (s x R R2, two-dimensional swimmer propels itself forward by coordinating body segments, emphasizing control over body dynamics in fluid-like environment.
C Hyperparameters
The setting of all hyperparameters for the training presented in Table
Table 3: Hyperparameters for training.
name
Value
Description
Optim
Adam
the optimizer of method
r_jttertion
8 x 10s
Maximum iteration steps until the end of training capacity of replay buffer
suffer_size
1 x 106
batch_size size
256
number samples from each update
evaluate cycle
10000
how often evaluate the model
No. of hidden layers
the number of hidden layers
No. of hidden nodes
256
the number of hidden nodes
Activation_n
GeLU
the activation of policy
Activation_22 2,
Mish
the activation of value distributions learning rate for policy learning rate for value distributions
Ire
3 x 10
tro:
3 x 10
0.005
target momentum coefficient discount factor
ry
0.99
T
10
diffusion step
16

---

D Details of the baseline methods
In this section, discuss details of the baseline methods with which compare method, including the implementation and hyperparameters of each algorithm.
DDPG
57 We use the high-quality implementations of reinforcement learning algorithms, i.e., [58]
OpenAl Baselines
with configuration basellines/dppgdapgg py evaluate the DDPG baseline on the MuJoCo benchmark.
PPO [55]
We use the code from the authors' repository
ggall
for their evaluation continuous control tasks, i.e., the MuloCo benchmark. We employ their configuration //tp_ppo_actttrrrraeeeeeeeeeeeea
TD3 22
We use the official code from
TD3
with hyper-parameters from main. py for the MuJoCo experiments.
SAC 23
We utilize the code from
with their configuration code/main. py for the MuloCo continuous control tasks in OpenAl gym, which is the PyTorch implementation of Soft Actor-Critic.
DSAC 224
We use the code from
httpg:///gtaabbbccm///gggignnnng
with their configuration example_train/mmiie wwwainaaaainnnnneeeeee. for the evaluation of MuJoCo continuous control tasks, which is the PyTorch implementation of Distributional Soft Actor-Critic.
DSACT 23]. We utilize the code from
http:s///gtttbb...cmiiiniggggggg
v2
with their configuration example train/dsacv2_mll http._mmmm___ooooomee_e offserial. py for the MuloCo benchmark, which is the PyTorch implementation of Distributional Soft Actor-Critic with Three Refinements.
SPO 56
We use the official code from
with haper-prammeters from mujoco/main py.
DIPO 4 We utilize the code from the authors' repository
attps://githie com comm/BellmaaTiiieittttoo
with hyper-parameters from main. py.
QSM (5). We provide a Py implementation of the QSM baseline based on the authors' repository, which is built top of re-implementation of the JAXRL framework
with hyper-parameterss from https://gghhaa.ooo/lleeerrreoeo.
QVPO (6). We use the official code from
17

---

https://githun http:/twwwwooccaoo//////////////
with hyper-parameters from main.p7 py.
DACER 77. For a fair comparison, we provide an implementation of the DACER method with PyTorch, which based on the official code (JAX) from
Diffoui-Dfffffuotttt 0n1 Mininne- RL
with hyper-parameters from scripta/train,mmjoc.
E Training and Inference Time
The training and inference time of VDRL and all baselines three tasks of MuJoCo, i.c., Ant-v3, Humanoid-v3, and Walker2d-v3, are shown in the Table 4 In particular, Notably, the official code of the QSM and DACER methods is based the JAX tramework, but the implementation of other baselines and our method based on PyTTorc.. In practice, the algorithm realized with JAX is 6-10 times faster than that realized with PyTTorch [6]. To ensure fair comparison, implementation of the QSM and DACER methods with PyTorch, which is based on the official code (JAX). Furthermore, the number diffusion steps for all diffusion-based online RL baselines (DIPO, QSM, QVPO, and DACER) is set to 20.
utilize the
Table 4: The training time (h) comparison on three tasks of MuJoCo Benchmarks.
VDRL
DIPO
QSM
QVPO
DACER
DSAC
DSACT
PPO
SAC
Ant-v3
7.9
11.1
10.2
9.1
9.8
5.3
5.8
0.4
3.8
Humanoid-v3
8.2
11.5
10.7
9.4
10.1
5.5
6.1
0.5
3.9
Walker2d-v3
7.8
11.0
9.9
8.9
9.6
5.2
5.7
0.4
3.8
Table 5: The inference time (ms) comparison on three tasks of MuJoCo Benchmarks.
VDRL
DIPO
QSM
QVPO
DACER
DSAC
DSACT
PPO
SAC
Ant-v3
3.6
5.8
6,4
6.2
5.4
1.9
2.1
0.2
0.4
Humanoid-v3
3.7
6.0
6.5
6.3
5.5
2.0
2.2
0.2
0.5
Walker2d-v3
3.6
5.7
6.3
6.1
5.3
1.9
2.1
0.2
0.4
From the results the above tables, we can observe:
The training and inference time of VDRL is faster than the diffusion-bassd online RL methods (i.c., DIPO, QSM, QVPO, and DACER), because the lower diffusion steps of VDRL (T=10) compared with diffusion-basee RL baselines (T=20) reduce the computational complexity.
All baasd RL methods are slower than distributional (i.e., DSAC and DSACT) or traditional (i.e.. SAC and PPO) ones, due to the iterative denoising process with some steps. Although VDRL is slower than the traditional or distributional RL methods inference, the inference times (3.7ms) are still tolerable in real-time application. As discussed in QVPO 66) most existing real robots only require a 50-Hz control policy, i.e., output action per 20 ms. Moreover, the training and inference time of VDRL can be further reduced by using the JAX framework it is necessary. like the official code QSM and DACER methods. Hence, the inference time is not bottleneck applying method to real-time applications.
Overall, the performance improvements of VDRL are relatively worth with appropriate longer training and inference time.
18

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificcton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
The contributions and scope of this paper are clearly delineated in the abstract
and introduction sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
19

---

Justification:
The limitations of this work are discussed in Section 6
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specifiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texi system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't "t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
The complete proof is deferred to Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The proposed algorithm of the training process is presented in Section 3.3
The intricate implementation details of and training hyperparameters are deferred to Section 5.2 and Appendix C
Guidelines:
20

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. Releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to hosted model (e.g., g., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The proposed algorithm of the training process is presented Section 3.3
The
intricate implementation details of and training hyperparameters are deferred to Section 5.2 and Appenddix C
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips.cm ec/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/tipps.cc/pullic/uuudee huwbbl///c/giddes//deeeeuuiii
/Ctd///oeeeeeeaeeaaaeee ssi onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
21

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The intricate implementation details of and training hyperparameters are
deferred Section 5.2 and Appendix Cj The more details of MuJoCo are introduced
in Appendix B
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All experimental results based on four training runs with different random seeds in the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative enror rates).
If error bars are reported in tables plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
22

---

Justification:
The information on computer resources needed to reproduce the experiments is provided in Section [5] Specifically, the all experiments are conducted on 2.90GHz Intel Core 17-10700 CPU, 64G RAM, and NVIDIA GeForce RTX 3090 GPU.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more computing than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics Ethics/https:/ http///wwpp..cccoooocccchccccaaa
Answer: [Yes]
Justification: In every respect, the research conducted in the paper conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper is foundational research and not tied to particular applications.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deploymeent of technologics that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
23

---

11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: This work cites the original paper that produced the code package. The details are deferred to Appendix
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paterssipppweiiiiee..... thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
There are no new assets in this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

---

The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMSs if it is important, original, or non-standard component of the core methods in this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development this research does not involve LLMS as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer LLM policy for what should or should not be described.
25


Reviews:
[{"review_id": "ohHtS2bphH", "reviewer": "Reviewer_yrGJ", "summary": "Accurate Q-value estimation is essential for effective policy optimization in reinforcement learning. This work introduces Value Diffusion Reinforcement Learning (VDRL), a model-free online RL approach that uses diffusion models to capture rich, multimodal value distributions. The key idea is a diffusion-based variational loss, theoretically shown to provide a tight lower bound under KL-divergence, leading to more precise value estimation. To further stabilize training, the authors propose a double value diffusion mechanism with sample selection. Experiments on MuJoCo benchmarks show that VDRL consistently outperforms strong state-of-the-art baselines, demonstrating its effectiveness.", "questions": "(1) Can the authors provide detailed runtime comparisons compared to the other baselines? \n\n(2) Can the authors expand the sensitivity analysis to at least two or three additional environments to confirm the consistency of conclusions?\n\n(3) How critical is the double value diffusion mechanism? Could the authors provide an ablation showing performance with and without this?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\nThe paper tackles an important problem, which is using diffusion models for value estimation in RL.\n\nThe method is motivated by theoretical analysis and results.\n\nThe experimental evaluation is rich and well executed comparing against 11 baseline algorithms on 8 tasks from Mujococ, reporting both plots and tables.\n\nThe paper is well-organized and well-written.\n\n**Weaknesses:**\n\nThe computational overhead of using diffusion in value estimation is not thoroughly analyzed. There is only a brief mention of trade-offs in the sensitivity analysis (Section 5.4) as well as a brief mention in the limitations of the computational overhead during online training, particularly in high-dimensional action space (Appendix E). Runtime analysis and computational complexity analysis are lacking.\n\nWhile MuJoCo is a representative, standard, and popular benchmark, it remains unclear how VDRL would perform in other tasks.\n\nSensitivity analysis was only conducted on Humanoid-v3 task as an example. However, it remains unclear if the same conclusions holds on the other tasks. The authors should provide ablations on different environments (say three of them) to confirm the observation.", "was_revised": true, "final_justification": "My concerns have been addressed, and I have decided to raise my score to 5. Please incorporate the additional experimental results and discussions into the revised version of the paper.", "rebuttal": "**We would like to thank you for recognizing our innovation and proposing some valuable comments. Below we make the responses to your concerns:**\n\n**Q1**: *The computational overhead of using diffusion in value estimation ..........*\n\n**A1**: Thanks for your question. The training and inference time of our method (VDRL) and other baselines on three tasks of MuJoCo, i.e., Ant-v3, Humanoid-v3, and Walker2d-v3, are shown in the following tables. **Notably, the official code of the QSM and DACER methods are based on the JAX framework but the implementation of other baselines and our method are based on pytorch.** In practice, the algorithm realized with jax is 6-10 times faster than that realized with pytorch [1]. To ensure a fair comparison, we utilize the implementation of the QSM and DACER methods with PyTorch, which is based on the official code (JAX). Furthermore, the number of diffusion steps for all diffusion-based online RL baselines (DIPO, QSM, QVPO, and DACER) is set to 20.\n\nTable 1: The training time (h) comparison on three tasks of MuJoCo Benchmarks.\n|          |                      |         | Diffusion-based |       |       |              |   *Distributional*    |           |  Traditional  |      |\n|:-------:|:-----------------:|:-------:|:-----------:|:-----:|:-----:|:--------------:|:--------------:|:-----:|:-------------:|:----:|\n|   Method   | **VDRL (ours)** | **DIPO** | **QSM**  | **QVPO** | **DACER** | *DSAC* | *DSACT* | PPO | TD3 | SAC |\n| Ant-v3      | 7.9        | 11.1 | 10.2 | 9.1  | 9.8   | 5.3  | 5.8   | 0.4 | 0.6 | 3.8 |\n| Humanoid-v3 | 8.2        | 11.5 | 10.7 | 9.4  | 10.1  | 5.5  | 6.1   | 0.5 | 0.6 | 3.9 |\n| Walker2d-v3 | 7.8        | 11.0 | 9.9  | 8.9  | 9.6   | 5.2  | 5.7   | 0.4 | 0.5 | 3.8 |\n\nTable 2: The inference time (ms) comparison on three tasks of MuJoCo Benchmarks.\n|          |                      |         | Diffusion-based |       |       |              |   *Distributional*    |           |  Traditional  |      |\n|:-------:|:-----------------:|:-------:|:-----------:|:-----:|:-----:|:--------------:|:--------------:|:-----:|:-------------:|:----:|\n|   Method   | **VDRL (ours)** | **DIPO** | **QSM**  | **QVPO** | **DACER** | *DSAC* | *DSACT* | PPO | TD3 | SAC |\n| Ant-v3      | 3.6 | 5.8  | 6.4  | 6.2  | 5.4   | 1.9  | 2.1   | 0.2 | 0.2 | 0.4 |\n| Humanoid-v3 | 3.7 | 6.0  | 6.5  | 6.3  | 5.5   | 2.0  | 2.2   | 0.2 | 0.3 | 0.5 |\n| Walker2d-v3 | 3.6 | 5.7  | 6.3  | 6.1  | 5.3   | 1.9  | 2.1   | 0.2 | 0.3 | 0.4 |\n\nFrom the results of above tables, we can observe:\n\n(i) **The training and inference time of our method is faster than the diffusion-based online RL methods** (i.e., DIPO, QSM, QVPO, and DACER), because the lower diffusion steps of VDRL (T=10) compared with diffusion-based online RL baselines (T=20) reduce the computational complexity; \n\n(ii) **All diffusion-based RL methods (i.e., DIPO, QSM, QVPO, DACER, VDRL) are slower than distributional (i.e., DSAC and DSACT) or traditional (i.e., SAC, TD3, and PPO) ones, due to the iterative denoising process with some steps.**  Although VDRL is slower than the traditional or distributional RL methods in inference, the inference times (3.7ms) is still tolerable in the real-time application. As discussed in QVPO [1], most existing real robots only require a 50-Hz control policy, i.e. output action per 20 ms. Moreover, the training and inference time of VDRL can be further reduced by using the JAX framework if it is necessary, like the official code of QSM and DACER methods. Hence, the inference time is not a bottleneck to applying our method to real-time applications.\n\n**Q2**: *While MuJoCo is a representative, standard, and popular benchmark, it remains unclear how VDRL would perform in other tasks.*\n\n**A2**: Thanks for your comments. Following your suggestion, here we conduct additional experiments on 10 inherently multimodal Atari games using the Arcade Learning Environment to verify the effectiveness of VDRL. The raw scores of these methods are shown in the Table 3, which demonstrate that **VDRL achieves better or competitive performance on 10 Atari games**.\n\nTable 3: The comparison of raw scores between VDRL and four baselines across ten Atari games, starting with 30 no-op actions, where the reference value from C51 [2 ], IQN [3 ], ER-DQN [4], and FQF [5].\n|      | Human   | C51    | IQN     | ER-DQN   | FQF | VDRL (ours) |\n|:------:|:-----:|:----:|:-------:|:------:|:-------:|:----:|\n| Alien    | 7127.7  | 3166   | 7022  | 6212 | 16754.6  | **16216**  |\n| Bowling    | 160.7   | 81.8   | 86.5  | 53.1  | 102.3    | **125.8**  |\n| Centipede  | 12017.0 | 9646   | 11561  | 22505.9  | 11526.0  | **26249.2**  |\n| Enduro    | 860.5   | 3454   | 2359   | 2339.5   | 2370.8   | **3278** |\n| Gopher    | 2412.5  | 33641  | 118365  | 115828.3 | 121144   | **121034.7**  |\n| Krull    | 2665.5  | 9735   | 10707   | 11318.5  | 10706.8  | **11652** |\n| Pong    | 14.6   | 20.9   | 21.0   | 21.0  | 21.0   | **22.1** |\n| Seaquest   | 42054.7 | 26434  | 30140   | 19401  | 29383.3  | **38061**  |\n| Tennis     | -8.3 | 23.1   | 23.6  | 5.8   | 22.6   | **27.4**  |\n| Venture  | 1187.5  | 1520   | 1318  | 1107  | 1112   | **1429**  |\n\n\n**Q3**: *Sensitivity analysis was only conducted on Humanoid-v3 task as an example. However, it remains unclear if the same conclusions holds on the other tasks. The authors should provide ablations on different environments (say three of them) to confirm the observation. Can the authors expand the sensitivity analysis to at least two or three additional environments to confirm the consistency of conclusions?*\n\n**A3**: Thanks for your comment. Following your suggestion, here we conduct additional experiments on three tasks of MuJoCo benchmark, including Ant-v3, Hopper-v3, and Walker2d-v3, as shown in Table 4 and 5. **These results further confirm the consistency of conclusions**: \n\n(i) the performance does not improve monotonically with an increasing number of diffusion steps, and the diffusion steps should be set to 10 by considering the trade-off between performance and computational complexity; \n\n(ii) the cosine and variance\\_preserve noise schedules perform similarly, and both outperform the linear schedule. Hence, we choose cosine schedule for all tasks.\n\nTable 4: The final performance of mean return and standard deviations for different diffusion steps.\n|                  |         T=5       |         T=10       |          T=20     |       T=30     |\n|:---------------:|:--------------:|:----------------:|:-------------:|:------------:|\n| Ant-v3        | 5824.41 ± 92.66   | **7236.67 ± 89.44**  | 6993.57 ± 92.96     | 6452.06 ± 97.20    |\n| Hopper-v3     | 3404.68 ± 9.92    | 3639.41 ± 8.73      | **3716.93 ± 13.58** | 3523.13 ± 11.05    |\n| Walker2d-v3   | 4938.82 ± 14.27   | **5216.24 ± 13.62** | 5208.74 ± 12.21     | 5127.44 ± 25.86    |\n\nTable 5: The final performance of mean return and standard deviations for different diffusion noise\nschedule.\n|                  | linear        | variance_preserve  | cosine        |\n|:------------:|:----------:|:----------:|:----------:|\n| Ant-v3       | 6422.53 ± 82.06   | 7119.02 ± 98.57    | **7236.67 ± 89.44**  |\n| Hopper-v3    | 3490.76 ± 8.17    | 3551.37 ± 10.19    | **3639.41 ± 8.73**   |\n| Walker2d-v3  | 4708.92 ± 12.84   | 5203.56 ± 13.99    | **5216.24 ± 13.62**  |\n\n**Q4**: *How critical is the double value diffusion mechanism? Could the authors provide an ablation showing performance with and without this?*\n\n**A4**: Thanks for your comments. To investigate the significance of the double value diffusion mechanism, here we conduct ablation experiments (as you suggested) on three tasks of MuJoCo benchmark, i.e., Ant-v3, Hopper-v3, and Walker2d-v3, as shown in the Table 6 and 7. The results show that VDRL with the double value diffusion mechanism achieves better performance than VDRL without this mechanism in terms of average returns and variance, which **verifies the effectiveness of the double value diffusion mechanism.**\n\nTable 6: The comparison of final performance between VDRL with and without the double value diffusion mechanism.\n|                 |         VDRL          | VDRL w/o the double value diffusion learning |\n|:------------:|:--------------------:|:--------------------:|\n| Ant-v3       | **7236.67 ± 89.44**               | 7113.44 ± 130.06                     |\n| Hopper-v3    | **3639.41 ± 8.73**                | 3526.17 ± 21.48                      |\n| Walker2d-v3  | **5216.24 ± 13.62**               | 5098.87 ± 22.86                      |\n\nTable 7: The comparison of average value estimation bias between VDRL with and without the double value diffusion learning rule.\n|                 | VDRL                      | VDRL w/o the double value diffusion learning |\n|:------------:|:----------------:|:----------------:|\n| Ant-v3       | **-7.36**        | 9.24             |\n| Hopper-v3    | **-113.70**      | 151.70           |\n| Walker2d-v3  | **-0.95**        | 12.09            |\n\n[1] Shutong Ding, et al. Diffusion-based reinforcement learning via q-weighted variational policy optimization. NIPS, 2024.\n\n[2] Marc G Bellemare, et al. A distributional perspective on reinforcement learning. ICML, 2017.\n\n[3] Will Dabney, et al. Implicit quantile networks for distributional reinforcement learning. ICML, 2018.\n\n[4] Mark Rowland, et al. Statistics and samples in distributional reinforcement learning. ICML, 2019.\n\n[5] Derek Yang, et al. Fully parameterized quantile function for distributional reinforcement learning. NIPS, 2019."}, {"review_id": "CU022c57eH", "reviewer": "Reviewer_WruH", "summary": "This paper introduces Value Diffusion Reinforcement Learning (VDRL) which employs diffusion models to learn expressive and multimodal returns in an off-policy distributional RL framework. The authors prove that a diffusion variational lower bound (VLB) is a tight lower bound on the KL divergence between the Bellman-updated target distribution and the learned diffusion critic (Theorem 1). They then build an algorithm incorporating mainstream RL techniques—namely a double-critic sampling method—to mitigate overestimation and stabilize training. Experimental results on MuJoCo locomotion tasks provide some statistical evidence that VDRL consistently outperforms baselines (PPO, TD3, SAC, DSAC, DSACT, and diffusion-based methods DIPO, QSM, QVPO, DACER) in both mean return and Q-estimation bias.", "questions": "1. Do MuJoCo locomotion tasks exhibit true multimodal return distributions? (any refs if yes)? Have you considered inherently multimodal benchmarks (e.g., Atari with stochastic rewards, hierarchical tasks, synthetic bimodal MDPs) to directly test your critic’s multimodal modeling claims? \n2. How does VDRL’s objective, derivation, and performance differ from QSM and QVPO? Please provide theoretical distinctions and empirical comparisons on a common benchmark.\n3. Could you consider reporting the per-iteration wall-clock time and total GPU-hours for VDRL vs. DSAC/DSACT.\n4. Will you consider testing VDRL under observation/action noise? If not, discuss anticipated failure modes and possible mitigation strategies.", "limitations": "* Compute Latency: Appendix E notes computational overhead but fails to provide concrete latency or throughput metrics, which are critical for high-frequency control applications.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 2, "originality": 1, "strengths_and_weaknesses": "1. Quality\n    * Strength: Theorem 1 cleanly links diffusion VLB to the RL KL-loss and gives a concrete algorithmic foundation.\n    * Weakness: Theorem 1 follows directly from standard DDPM variational bounds (e.g., Eq. 5 in ref [41]), thus offering limited new insight. As part of their theoretical exploration it would have been nice to see an analysis of Bellman-operator contraction/convergence (like in C51 [Bellemare et al. 2017], IQN [Dabney et al. 2018]), although this is not strictly necessary as long as learning curves show stability and convergence (which they do), but this brings me to the next point which is the very low number of seeds (4) used to make statistical arguments of the superiority of the method. As a rule of thumb, having 10 seeds is a requirement to make such statement; any less would be questionable so I would not take their empirical tests seriously unless they fix it in the rebuttal. I looked at one of the main baselines (QVPO) [1], and there doesn’t seems to be consistent final returns between the two papers of the algorithm QVPO judging by Figure 1 in your paper and Figure 3 in [1].\n2. Clarity\n    * Strength: Well-motivated exposition with clear pseudocode (Algorithm 1).\n    * Weakness: Minor typos (e.g., “introducs”, “alogirthms”); Eq. 10 needs a revision. Hyperparameters (diffusion steps, noise schedules, seed count) are scattered across the main text, Section 5.4, and Appendix D instead of centralized.\n3. Significance\n    * Strength: Addresses the limitation of unimodal Gaussian or quantile critics by modeling full return distributions, leading to better representation of complex reward landscapes.\n    * Weakness: Evaluation is confined to deterministic MuJoCo environments; lacks tests under sensor/motor noise, partial observability, or on real-world robotic platforms. The paper also omits reporting compute–performance tradeoffs (e.g., reverse-sampling cost, wall-clock metrics).\n4. Originality\n    * Strength: First to apply diffusion generative modeling directly to value distributions in RL rather than to policies.\n    * Weakness: The work overlaps heavily with recent diffusion-value works (Mazoure et al. 2023; QSM; QVPO). The core DDPM VLB machinery is reused rather than extended, so the conceptual novelty is incremental.\n\n\nRef:\n[1] Ding, S. et al . Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization. NeurIPS 2024.", "was_revised": true, "final_justification": "Thank you for addressing most of my points (improving seed count and the benchmark multimodality (the 10 Atari examples)). I think the paper is well-executed at this point given the ideas proposed, but the contribution (in terms of the conceptual novelty and possible significance) is still, in my opinion, incremental. To recognize the efforts and the new results you presented in the rebuttal, I will raise my score to 4.", "rebuttal": "**We would like to thank you for proposing some valuable comments. Below we respond to your concerns:**\n\n**Weakness 1**\n\n**A**: Thanks for your comments. We make the clarification as below:\n\n(i) **The core theoretical contribution of Theorem 1 lies in establishing the formal connection between the Variational Bound Objective (VBO) of diffusion models and the Value Distribution Learning Objective (VDLO) in distributional RL,** inspired by DDPM variational bounds but different from it. i) solved question is different; ii) The whole derivation process is different from the derivation of DDPM variational bounds; iii) The style of variational bounds, such as DDPM, VAE and ours, look similar, which might be the understanding you think \"Theorem 1 follows directly from standard DDPM variational bounds\". \n\n(ii) We conduct experiments to compare VDRL and baselines on three tasks of MuJoCo benchmark with **10 runs and different random seeds**. The Table 1 demonstrates the consistent conclusion claimed in the manuscript.\n\nTable 1: The final performance of VDRL and baselines on three tasks of MuJoCo benchmark with\nten seeds over 800,000 training steps.\n|      | SAC     | QVPO  | DSACT  | VDRL (ours)  |\n|:----------:|:-------:|:--------:|:-------:|:------:|\n| Ant-v3           | 4286.35 ± 135.19  | 5230.64 ± 80.32   | 5314.06 ± 141.58  | **7368.67 ± 81.27**  |\n| Halfcheetah-v3   | 10308.47 ± 90.84  | 9104.28 ± 76.11   | 12069.31 ± 152.02 | **12643.09 ± 156.44** |\n| Walker-v3        | 4493.60 ± 64.31   | 4197.09 ± 47.77   | 5114.26 ± 32.15   | **5340.83 ± 24.95**   |\n\n(iii) For the inconsistency of results between our work and QVPO, we make the clarification that **all comparison results of QVPO on MuJoCo benchmark are strictly produced from the official code (https://github.com/wadx2019/qvpo).**\n\n**Weakness 2**\n\n**A**: Thanks for your review. We make sure to correct these typos and summarize these hyperparameters  into Table 3 of manuscript in the final version, according to your suggestion.\n\n**Weakness 3 (Part 1) and Question 4**\n\n**A**: Thanks for your comments. The MuJoCo environments have the standard protocols for evaluating various algorithms. Previous online RL methods follows the protocols. For fair comparisons, we also take the same configurations. We very appreciate these points you suggested, which is interesting and promising future direction. But they seems beyond the scope of this work. Due to time limitation, here we conduct an experiment with adding some noises to observation. We observe that noise can disrupt value learning in some tasks by distorting state-action representations. Techniques like data augmentation or adversarial training could improve noise resilience, and we'll explore these ideas in future work. \n\n**Weakness 3 (Part 2)**\n\n**A**: Thanks for your question. The training and inference time of our method and other baselines on three tasks of MuJoCo, are shown in the following tables. **Notably, the official code of the QSM and DACER methods are based on the JAX framework but the implementation of other baselines and our method are based on pytorch.** In practice, the algorithm realized with jax is 6-10 times faster than that realized with pytorch[1]. To ensure a fair comparison, we utilize the implementation of the QSM and DACER methods with PyTorch, which is based on the official code (JAX). \n\nTable 2: The training time (h) comparison on three tasks of MuJoCo Benchmarks.\n|          |           |         | Diffusion-based |       |       |         |   *Distributional*    |       |  Traditional  |      |\n|:-------:|:----------:|:-------:|:-----------:|:-----:|:-----:|:--------------:|:--------------:|:-----:|:-------:|:----:|\n|   Method   | **VDRL (ours)** | **DIPO** | **QSM**  | **QVPO** | **DACER** | *DSAC* | *DSACT* | PPO | TD3 | SAC |\n| Ant-v3      | 7.9    | 11.1 | 10.2 | 9.1  | 9.8   | 5.3  | 5.8   | 0.4 | 0.6 | 3.8 |\n| Humanoid-v3 | 8.2   | 11.5 | 10.7 | 9.4  | 10.1  | 5.5  | 6.1   | 0.5 | 0.6 | 3.9 |\n| Walker2d-v3 | 7.8 | 11.0 | 9.9  | 8.9  | 9.6   | 5.2  | 5.7   | 0.4 | 0.5 | 3.8 |\n\nTable 3: The inference time (ms) comparison on three tasks of MuJoCo Benchmarks.\n|    |     |      | Diffusion-based |      |      |      |   *Distributional*  |    |  Traditional  |     |\n|:-------:|:-------:|:-------:|:------:|:-----:|:-----:|:---------:|:-------:|:-----:|:----------:|:----:|\n|   Method   | **VDRL (ours)** | **DIPO** | **QSM**  | **QVPO** | **DACER** | *DSAC* | *DSACT* | PPO | TD3 | SAC |\n| Ant-v3      | 3.6 | 5.8  | 6.4  | 6.2  | 5.4   | 1.9  | 2.1   | 0.2 | 0.2 | 0.4 |\n| Humanoid-v3 | 3.7 | 6.0  | 6.5  | 6.3  | 5.5   | 2.0  | 2.2   | 0.2 | 0.3 | 0.5 |\n| Walker2d-v3 | 3.6 | 5.7  | 6.3  | 6.1  | 5.3   | 1.9  | 2.1   | 0.2 | 0.3 | 0.4 |\n\nFrom the results of above tables, we can observe:\n\n(i) **The training and inference time of our method is faster than the diffusion-based online RL methods** (i.e., DIPO, QSM, QVPO, and DACER), because the lower diffusion steps of VDRL (T=10) compared with diffusion-based online RL baselines (T=20) reduce the computational complexity; \n\n(ii) **All diffusion-based RL methods (i.e., DIPO, QSM, QVPO, DACER, VDRL) are slower than distributional (i.e., DSAC and DSACT) or traditional (i.e., SAC, TD3, and PPO) ones, due to the iterative denoising process with some steps.**  Although VDRL is slower than the traditional or distributional RL methods in inference, the inference times (3.7ms) is still tolerable in the real-time application. As discussed in QVPO, most existing real robots only require a 50-Hz control policy, i.e. output action per 20 ms. Moreover, the training and inference time of VDRL can be further reduced by using the JAX framework if it is necessary, like the official code of QSM and DACER methods. \n\n**Overall, the performance improvements of our method are relatively worth with appropriate longer training and inference time.**\n\n**Weakness 4**\n\n**A**: Thanks for your comments. We make the clarification as follows:\n\n(i) **Main difference**: Our work  is to derive and build **value-distribution diffusion model**, while  these works (QSM and QVPO you mentioned) falls in the category of **policy-based diffusion models**.  That is, VDRL is fundamentally different from these diffusion-based policy RL methods you said.\n\n(ii) The core conceptual novelty of our work lies in **establishing the (both theoretical and methodological) connection between the Variational Bound Objective (VBO) in diffusion models and the Value Distribution Learning Objective (VDLO) in distributional RL**. Crucially, we **theoretical prove** that the VBO serves as a **tight lower bound** of the VDLO under KL-divergence minimization. This theoretical foundation underpins our methodological contribution.\n\n**Question 1**\n\n**A**: Thanks for your comments. We make the clarification as below:\n\n(i) The MuJoCo locomotion benchmark is widely-used by the existing online RL methods to verify their performance. In practice, we experimentally fit the values on this benchmark, observe that the value distribution is not unimodal.\n\n(ii) We conduct experiments on 10 inherently multimodal Atari games to verify the effectiveness of VDRL compared with four methods, as shown in Table 4. The results demonstrate that **VDRL achieves better or competitive performance on ten Atari games.**\n\nTable 4: The comparison of raw scores between VDRL and four baselines across ten Atari games, starting with 30 no-op actions.\n|      | Human   | C51    | IQN     | ER-DQN   | FQF | VDRL (ours) |\n|:------:|:-----:|:----:|:-------:|:------:|:-------:|:----:|\n| Alien    | 7127.7  | 3166   | 7022  | 6212 | 16754.6  | **16216**  |\n| Bowling    | 160.7   | 81.8   | 86.5  | 53.1  | 102.3    | **125.8**  |\n| Centipede  | 12017.0 | 9646   | 11561  | 22505.9  | 11526.0  | **26249.2**  |\n| Enduro    | 860.5   | 3454   | 2359   | 2339.5   | 2370.8   | **3278** |\n| Gopher    | 2412.5  | 33641  | 118365  | 115828.3 | 121144   | **121034.7**  |\n| Krull    | 2665.5  | 9735   | 10707   | 11318.5  | 10706.8  | **11652** |\n| Pong    | 14.6   | 20.9   | 21.0   | 21.0  | 21.0   | **22.1** |\n| Seaquest   | 42054.7 | 26434  | 30140   | 19401  | 29383.3  | **38061**  |\n| Tennis     | -8.3 | 23.1   | 23.6  | 5.8   | 22.6   | **27.4**  |\n| Venture  | 1187.5  | 1520   | 1318  | 1107  | 1112   | **1429**  |\n\n**Question 2**\n\n**A**: Thanks for your comments. We clarify the distinctions as follows:\n\n(i) **Theoretical objective and derivation**: Our optimization objective is derived from **the value distribution learning** (i.e., Theorem 1 of Section 3.1 in the original manuscript), while the objective of QSM is **the alignment of the score of learning policy**. The difference of derivation and formulation for our method and QVPO **was introduced in Remark 1 (Line 154-160)** of original manuscript. \n\n(ii) **Empirical Comparison**: The empirical comparison of VDRL, QSM and QVPO on MuJoCo benchmark **have been provided in Figure 1 and Table 1 of original manuscript.** \n\n**Question 3**\n\n**A**: We conduct experiments to compare VDRL, DSAC, and DSACT on three tasks of MuJoCo benchmark with the function of wall clock time.\n\n(i) The Table 5 further demonstrates that VDRL achieve better or competitive performance compared with these methods.\n\nTable 5 The final performance of VDRL and baselines on three tasks of MuJoCo benchmark over 8 hours training time.\n|     | DSAC   | DSACT  | VDRL (ours)  |\n|:-------:|:------:|:-------:|:------:|\n| Ant-v3    | 4625.94 ± 226.31  | 5784.20 ± 105.59  | **7248.18 ± 89.14**  |\n| Halfcheetah-v3   | 11698.55 ± 136.47 | 11740.82 ± 130.08 | **12840.61 ± 120.97** |\n| Walker2d-v3      | 4913.82 ± 184.09  | 5167.18 ± 29.67   | **5248.36 ± 11.56**   |\n\n(ii) For the comparison of total GPU-hours (training time) for VDRL and these baselines , please refer to the answer of Weakness 3 (Part 2).\n\n[1] Shutong Ding, et al. Diffusion-based reinforcement learning via q-weighted variational policy optimization. NIPS, 2024."}, {"review_id": "ciplWyJos7", "reviewer": "Reviewer_KCCf", "summary": "This paper aims to address the distributional bias in value distributions via diffusion models (multi-modal value distributions). Specifically, the proposed variational loss of diffusion-based value distribution is theoretically proven to be the lower bound of the optimization objective.\nExperimental results on the MuJoCo benchmark demonstrate that the proposed VDRL outperforms existing baselines significantly.", "questions": "### Questions\n1. Could the authors provide the training and inference speed comparison? One of my concerns is the computational budget of the diffusion value function.\n2. Experimental results on value estimation accuracy.\n3. Experimental results on discrete control tasks (e.g., Atari-10). Comparison of the proposed diffusion value method with the IQN.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n1. It is an interesting idea to integrate the diffusion model and distributional RL.\n2. The writing is clear, but the structure should be improved.\n\n### Weakness\n1. Lines 171-174: The statements of underestimation and overestimation are not correct. These estimation biases have different effects in different task settings [1]. Additionally, it is possible to control the estimation bias via maxmin value estimation.\n2. There are no related experiments of value estimation accuracy that validate the paper's statements: \"utilizing the generative capacity of diffusion models to represent expressive and multimodal value distribution\"\n\n\n### Minor\n1. For coherency of reading, the related work section is usually placed after the introduction section or before the conclusion section.\n\n[1] Lan, Qingfeng, et al. \"Maxmin q-learning: Controlling the estimation bias of q-learning.\" arXiv preprint arXiv:2002.06487 (2020).", "was_revised": true, "final_justification": "I have carefully read the response, other reviews, and the paper. This paper has a novel motivation for advancing RL with the diffusion model, and comprehensive experiments demonstrating the superior performance. Thus, I tend to raise my score to 4.", "rebuttal": "**We would like to thank you for recognizing our innovation and proposing some valuable comments. Below we make the responses to your concerns:**\n\n**Q1**: *Lines 171-174: The statements of underestimation and overestimation are not correct. These estimation biases have different effects in different task settings [1]. Additionally, it is possible to control the estimation bias via maxmin value estimation.*\n\n**A1**: Thanks for your comments. We make the clarification as follows:\n\n(i) For the first sentence of lines 171-174, we conduct additional experiments to verify the effectiveness of the double value diffusion learning rule on three tasks of MuJoCo benchmark, i.e., Ant-v3, Hopper-v3, and Walker2d-v3, as shown in the Table 1 and 2. The result of Table 1 indicates that **this learning rule can further reduce value overestimation, and induce a relatively minor value underestimation in some tasks.** \n\n(ii) From the experimental results, we can observe that the slight value underestimation induced by this learning rule could improve the performance on the MuJoCo benchmark, which may because **the underestimated value do not propagate explicitly through policy update during training.** Besides, **the effect of underestimation and overestimation**, i.e., the second sentence of lines 171-174, **have also been similarly discussed in TD3 [1] and DSACT [2]**. \n\n(iii) As you suggested and introduced in Maxmin Q-learning [3], there are different effects of these estimation bias for distinct tasks. Thus, one promising future direction is to integrate the technique (maxmin value estimation) for different environment settings.\n\nTable 1: The comparison of average value estimation bias between VDRL with and without the double value diffusion learning rule.\n|                 | VDRL                      | VDRL w/o the double value diffusion learning |\n|:------------:|:----------------:|:----------------:|\n| Ant-v3       | **-7.36**        | 9.24             |\n| Hopper-v3    | **-113.70**      | 151.70           |\n| Walker2d-v3  | **-0.95**        | 12.09            |\n\nTable 2: The comparison of final performance between VDRL with and without the double value diffusion mechanism.\n|                 |         VDRL          | VDRL w/o the double value diffusion learning |\n|:------------:|:--------------------:|:--------------------:|\n| Ant-v3       | **7236.67 ± 89.44**    | 7113.44 ± 130.06        |\n| Hopper-v3    | **3639.41 ± 8.73**      | 3526.17 ± 21.48       |\n| Walker2d-v3  | **5216.24 ± 13.62**    | 5098.87 ± 22.86    |\n\n**Q2**: *There are no related experiments of value estimation accuracy that validate the paper's statements: \"utilizing the generative capacity of diffusion models to represent expressive and multimodal value distribution\". Experimental results on value estimation accuracy.*\n\n**A2**: We make a clarification for the question you concerned. **The experimental results of value estimation accuracy were provided in Table 2 of the original manuscript.** The table shows the average value estimation bias, i.e., the difference between the estimated Q-value and the true Q-value, for VDRL, traditional model-free online RL, and distributional RL methods. The results show that **VDRL consistently achieves lower bias than DASC and DSACT, highlighting the advantages of leveraging diffusion models to capture multimodal and complex value distributions for improved estimation accuracy.**\n\n**Q3**: *For coherency of reading, the related work section is usually placed after the introduction section or before the conclusion section.*\n\n**A3**: Thanks for your suggestion. To improve the reading coherence, we will adjust the related work section to precede the conclusion section in the final version.\n\n**Q4**: *Could the authors provide the training and inference speed comparison? One of my concerns is the computational budget of the diffusion value function.*\n\n**A4**: Thanks for your question. The training and inference time of our method (VDRL) and other baselines on three tasks of MuJoCo, i.e., Ant-v3, Humanoid-v3, and Walker2d-v3, are shown in the following tables. **Notably, the official code of the QSM and DACER methods are based on the JAX framework but the implementation of other baselines and our method are based on pytorch.** In practice, the algorithm realized with jax is 6-10 times faster than that realized with pytorch [4]. To ensure a fair comparison, we utilize the implementation of the QSM and DACER methods with PyTorch, which is based on the official code (JAX). Furthermore, the number of diffusion steps for all diffusion-based online RL baselines (DIPO, QSM, QVPO, and DACER) is set to 20.\n\nTable 3: The training time (h) comparison on three tasks of MuJoCo Benchmarks.\n|          |           |         | Diffusion-based |       |       |         |   *Distributional*    |       |  Traditional  |      |\n|:-------:|:----------:|:-------:|:-----------:|:-----:|:-----:|:--------------:|:--------------:|:-----:|:-------:|:----:|\n|   Method   | **VDRL (ours)** | **DIPO** | **QSM**  | **QVPO** | **DACER** | *DSAC* | *DSACT* | PPO | TD3 | SAC |\n| Ant-v3      | 7.9    | 11.1 | 10.2 | 9.1  | 9.8   | 5.3  | 5.8   | 0.4 | 0.6 | 3.8 |\n| Humanoid-v3 | 8.2   | 11.5 | 10.7 | 9.4  | 10.1  | 5.5  | 6.1   | 0.5 | 0.6 | 3.9 |\n| Walker2d-v3 | 7.8 | 11.0 | 9.9  | 8.9  | 9.6   | 5.2  | 5.7   | 0.4 | 0.5 | 3.8 |\n\nTable 4: The inference time (ms) comparison on three tasks of MuJoCo Benchmarks.\n|    |     |      | Diffusion-based |      |      |      |   *Distributional*  |    |  Traditional  |     |\n|:-------:|:-------:|:-------:|:------:|:-----:|:-----:|:---------:|:-------:|:-----:|:----------:|:----:|\n|   Method   | **VDRL (ours)** | **DIPO** | **QSM**  | **QVPO** | **DACER** | *DSAC* | *DSACT* | PPO | TD3 | SAC |\n| Ant-v3      | 3.6 | 5.8  | 6.4  | 6.2  | 5.4   | 1.9  | 2.1   | 0.2 | 0.2 | 0.4 |\n| Humanoid-v3 | 3.7 | 6.0  | 6.5  | 6.3  | 5.5   | 2.0  | 2.2   | 0.2 | 0.3 | 0.5 |\n| Walker2d-v3 | 3.6 | 5.7  | 6.3  | 6.1  | 5.3   | 1.9  | 2.1   | 0.2 | 0.3 | 0.4 |\n\nFrom the results of above tables, we can observe:\n\n(i) **The training and inference time of our method is faster than the diffusion-based online RL methods** (i.e., DIPO, QSM, QVPO, and DACER), because the lower diffusion steps of VDRL (T=10) compared with diffusion-based online RL baselines (T=20) reduce the computational complexity; \n\n(ii) **All diffusion-based RL methods (i.e., DIPO, QSM, QVPO, DACER, VDRL) are slower than distributional (i.e., DSAC and DSACT) or traditional (i.e., SAC, TD3, and PPO) ones, due to the iterative denoising process with some steps.**  Although VDRL is slower than the traditional or distributional RL methods in inference, the inference times (3.7ms) is still tolerable in the real-time application. As discussed in QVPO [1], most existing real robots only require a 50-Hz control policy, i.e. output action per 20 ms. Moreover, the training and inference time of VDRL can be further reduced by using the JAX framework if it is necessary, like the official code of QSM and DACER methods. Hence, the inference time is not a bottleneck to applying our method to real-time applications.\n\n**Q5**: *Experimental results on discrete control tasks (e.g., Atari-10). Comparison of the proposed diffusion value method with the IQN.*\n\n**A5**: According to your suggestion, we conduct experiments on 10 Atari games using the Arcade Learning Environment to compare our method with the IQN [5]. The raw scores of VDRL and IQN are shown in the Table 5, which demonstrate that **our proposed diffusion value method achieves better or competitive performance** on ten Atari games.\n\nTable 5. The comparison of raw scores between VDRL and IQN across ten Atari games, starting with 30 no-op actions, where the reference value from IQN [5].\n| Game       | Human     | IQN      | VDRL (ours)  |\n|:----------:|:--------:|:-------:|:------------:|\n| Alien      | 7127.7   | 7022    | **16216**    |\n| Bowling    | 160.7    | 86.5    | **125.8**    |\n| Centipede  | 12017.0  | 11561   | **26249.2**  |\n| Enduro     | 860.5    | 2359    | **3278**     |\n| Gopher     | 2412.5   | 118365  | **121034.7** |\n| Krull      | 2665.5   | 10707   | **11652**    |\n| Pong       | 14.6     | 21.0    | **22.1**     |\n| Seaquest   | 42054.7  | 30140   | **38061**    |\n| Tennis     | -8.3     | 23.6    | **27.4**     |\n| Venture    | 1187.5   | 1318    | **1429**     |\n\n[1] Scott Fujimoto, et al. Addressing function approximation error in actor-critic methods. ICML, 2018.\n\n[2] Jingliang Duan, et al. stributional soft actor-critic with three refinements. TPAMI, 2025.\n\n[3] Qingfeng Lan, et al. Maxmin q-learning: Controlling the estimation bias of q-learning. ICML, 2020\n\n[4] Shutong Ding, et al. Diffusion-based reinforcement learning via q-weighted variational policy optimization. NIPS, 2024.\n\n[5] Will Dabney, et al. Implicit quantile networks for distributional reinforcement learning. ICML, 2018."}, {"review_id": "0FJLMp5N1f", "reviewer": "Reviewer_WsdU", "summary": "This paper proposes Value Diffusion Reinforcement Learning (VDRL), a model-free, off-policy reinforcement learning algorithm that employs diffusion models to represent value distributions. To further improve stability, they propose double value diffusion learning with sample selection, inspired by double Q-learning. VDRL is evaluated on eight MuJoCo benchmark tasks and compared against eleven baseline RL methods spanning traditional, distributional, and diffusion-policy RL paradigms.", "questions": "- Can you provide wall-clock training and inference times per step for VDRL compared to baselines?\n- After changing the x-axis of Figure 1 to wall time, does VDRL still retain its advantage?\n- How does VDRL compare to the quantile-based methods [27–32]?\n- I noticed that the performance of your DACER is quite different from that in the original paper. The distributed Q learning part based on DACER is the same as DSACT, but the performance in the paper is lower than DSACT. Can you run the experiment using the jax version of the code?\n- What are the differences between your work and the accepted paper (Liu et al, ITSC 2025)?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- Addressing Q-value estimation bias is a core challenge in RL. VDRL has the potential to significantly impact future work in distributional RL.\n- Value estimation bias is quantitatively analyzed, which is an uncommon but insightful inclusion.\n- The paper makes a compelling case for moving beyond unimodal Gaussian value approximations, providing both empirical and conceptual justification.\n\nWeaknesses:\n- The proof in Line 500 relies on the assumption that the policy has fully converged (the authors do not mention this in Theorem 1). However, in practice, the policy is extremely hard to fully converge, especially when using a neural network as the policy. Therefore, $\\frac{P_{\\theta}(Y_{1:T} \\mid s, a, Y_0)}{q(Y_{1:T} \\mid s, a, Y_0)}=1$ in Line 500 does not hold in general, which means Theorem 1 does not hold.\n- There are no concrete data for training and inference time. The training and inference processes of diffusion models are notably time-consuming. Whether the performance improvements are worth such a high computational cost is debatable.\n- The x-axis in Figure 1 should be changed from training steps to wall time; otherwise, the comparison is unfair because diffusion models need more time for training in general.\n- Line 40 mentions quantile-based methods [27–32], which are also multimodal value distributions, but no experiments are conducted to compare against these baselines.\n\nMoreover, I just noticed a paper [1] that has already been accepted but is not cited in your work. Its background and method are highly similar to yours. Both use diffusion models to fit multimodal value networks, and the loss functions are identical—i.e., Equation (10) in your paper and Equation (15) in [1]. Could you clarify the differences between your work and this accepted paper?\n\n[1] Tong Liu, et al. Distributional Soft Actor-Critic with Diffusion Policy. ITSC 2025. ([arXiv](https://arxiv.org/abs/2507.01381))", "was_revised": true, "final_justification": "This work did not achieve the claimed state-of-the-art results, and the performance of the compared algorithms was biased. For example, DACER was based on DSACT, but its performance was worse than it.", "rebuttal": "**We would like to thank you for proposing some valuable comments. Below we respond to your concerns:**\n\n**Q1**: *The proof in Line 500  ..........*\n\n**A1**: Thanks for your comments. We make the clarification as follows:\n\n(i) The assumption of proof in Line 500 of manuscript is **the learning policy converges, which has been presentation in Line 152 (Theorem 1) of original manuscript  (i.e., the equality holds when the policy converges).**\n\n(ii) For the question (difficult to converge) you concerned, we experimentally observed that it **works well** on convergence and performance, as shown in Figure 1 and Table 1 of original manuscript. It is noted that the formula $\\frac{P_{\\theta}(Y_{1:T}|s,a,Y_0)}{q(Y_{1:T}|s,a,Y_0)}=1$ is the tightest case (i.e., an idea lower bound), which likes the extreme case of lower bound of Variational auto-encoder (VAE). In other work, the tightest case (w.r.t lower bound) does not affect the validity of this theorem, in analogy to the bound theory of VAE.\n\n**Q2**: *There are no concrete data for .....*\n\n**A2**: Thanks for your question. The training and inference time of our method (VDRL) and other baselines on three tasks of MuJoCo, i.e., Ant-v3, Humanoid-v3, and Walker2d-v3, are shown in the following tables. **Notably, the official code of the QSM and DACER methods are based on the JAX framework but the implementation of other baselines and our method are based on pytorch.** In practice, the algorithm realized with jax is 6-10 times faster than that realized with pytorch[1]. To ensure a fair comparison, we utilize the implementation of the QSM and DACER methods with PyTorch, which is based on the official code (JAX). Furthermore, the number of diffusion steps for all diffusion-based online RL baselines (DIPO, QSM, QVPO, and DACER) is set to 20.\n\nTable 1: The training time (h) comparison on three tasks of MuJoCo Benchmarks.\n|       |      |       | Diffusion-based |       |       |         |   *Distributional*    |       |  Traditional  |      |\n|:-------:|:-----:|:-------:|:-------:|:-----:|:-----:|:------:|:------:|:-----:|:-----:|:----:|\n|   Method   | **VDRL (ours)** | **DIPO** | **QSM**  | **QVPO** | **DACER** | *DSAC* | *DSACT* | PPO | TD3 | SAC |\n| Ant-v3      | 7.9    | 11.1 | 10.2 | 9.1  | 9.8   | 5.3  | 5.8   | 0.4 | 0.6 | 3.8 |\n| Humanoid-v3 | 8.2   | 11.5 | 10.7 | 9.4  | 10.1  | 5.5  | 6.1   | 0.5 | 0.6 | 3.9 |\n| Walker2d-v3 | 7.8 | 11.0 | 9.9  | 8.9  | 9.6   | 5.2  | 5.7   | 0.4 | 0.5 | 3.8 |\n\nTable 2: The inference time (ms) comparison on three tasks of MuJoCo Benchmarks.\n|    |     |      | Diffusion-based |      |      |      |   *Distributional*  |    |  Traditional  |     |\n|:-------:|:-------:|:-------:|:------:|:-----:|:-----:|:----:|:-------:|:-----:|:-----:|:----:|\n|   Method   | **VDRL (ours)** | **DIPO** | **QSM**  | **QVPO** | **DACER** | *DSAC* | *DSACT* | PPO | TD3 | SAC |\n| Ant-v3      | 3.6 | 5.8  | 6.4  | 6.2  | 5.4   | 1.9  | 2.1   | 0.2 | 0.2 | 0.4 |\n| Humanoid-v3 | 3.7 | 6.0  | 6.5  | 6.3  | 5.5   | 2.0  | 2.2   | 0.2 | 0.3 | 0.5 |\n| Walker2d-v3 | 3.6 | 5.7  | 6.3  | 6.1  | 5.3   | 1.9  | 2.1   | 0.2 | 0.3 | 0.4 |\n\nFrom the results of above tables, we can observe:\n\n(i) **The training and inference time of our method is faster than the diffusion-based online RL methods** (i.e., DIPO, QSM, QVPO, and DACER), because the lower diffusion steps of VDRL (T=10) compared with diffusion-based online RL baselines (T=20) reduce the computational complexity; \n\n(ii) **All diffusion-based RL methods (i.e., DIPO, QSM, QVPO, DACER, VDRL) are slower than distributional (i.e., DSAC and DSACT) or traditional (i.e., SAC, TD3, and PPO) ones, due to the iterative denoising process with some steps.**  Although VDRL is slower than the traditional or distributional RL methods in inference, the inference times (3.7ms) is still tolerable in the real-time application. As discussed in QVPO, most existing real robots only require a 50-Hz control policy, i.e. output action per 20 ms. Moreover, the training and inference time of VDRL can be further reduced by using the JAX framework if it is necessary, like the official code of QSM and DACER methods. Hence, the inference time is not a bottleneck to applying our method to real-time applications.\n\n**Overall, the performance improvements of our method are relatively worth with appropriate longer training and inference time.**\n\n**Q3**: *The x-axis in Figure 1 should be ...... Can you provide wall-clock training ..........? After changing the x-axis of Figure 1 to wall time, ...*\n\n**A3**: Thanks for your comments. We make the clarification as follows:\n\n(i) **The setting of x-axis (training steps) in Figure 1 of our work is the same as the existing diffusion-based online RL methods as well as the traditional online RL methods**, including DIPO, QSM, and QVPO, SAC. So the comparison of performance for VDRL and baselines are reasonable and fair. \n\n(ii) For the comparison of training and inference time for VDRL and baselines, please refer to the A2.\n\n(iii) According to your suggestion, here we conduct experiments to compare VDRL and baselines on three tasks of MuJoCo benchmark with the function of wall clock time. Table 3 shows the mean returns and standard deviations for our method (VDRL) and baselines, which further demonstrate that VDRL achieve better or competitive performance compared with these baselines.\n\nTable 3 The final performance of VDRL and baselines on three tasks of MuJoCo benchmark over 8 hours training time.\n|           | SAC    | QVPO     | DSACT   | VDRL (ours)       |\n|:------:|:-------:|:-----:|:--------:|:---------:|\n| Ant-v3           | 4386.35 ± 50.42   | 5324.96 ± 83.41   | 5784.20 ± 105.59  | **7248.18 ± 89.14**  |\n| Halfcheetah-v3   | 10294.37 ± 105.73 | 9013.16 ± 77.42   | 11740.82 ± 130.08 | **12840.61 ± 120.97** |\n| Walker-v3        | 4669.25 ± 38.11   | 4398.74 ± 46.01   | 5167.18 ± 29.67   | **5248.36 ± 11.56**   |\n\n**Q4**: *Line 40 mentions quantile-based methods, ........?*\n\n**A4**: Thanks for your comments. Following your suggestion, here we conduct experiments on ten Atari games using the Arcade Learning Environment to compare our method with four quantile-based methods, including C51, IQN, ER-DQN, and FQF. The raw scores of these methods are shown in the Table 4, which demonstrate that **VDRL achieves better or competitive performance on ten Atari games.**\n\nTable 3: The comparison of raw scores between VDRL and four baselines across ten Atari games, starting with 30 no-op actions.\n|      | Human   | C51    | IQN     | ER-DQN   | FQF | VDRL (ours) |\n|:------:|:-----:|:----:|:-------:|:------:|:-------:|:----:|\n| Alien    | 7127.7  | 3166   | 7022  | 6212 | 16754.6  | **16216**  |\n| Bowling    | 160.7   | 81.8   | 86.5  | 53.1  | 102.3    | **125.8**  |\n| Centipede  | 12017.0 | 9646   | 11561  | 22505.9  | 11526.0  | **26249.2**  |\n| Enduro    | 860.5   | 3454   | 2359   | 2339.5   | 2370.8   | **3278** |\n| Gopher    | 2412.5  | 33641  | 118365  | 115828.3 | 121144   | **121034.7**  |\n| Krull    | 2665.5  | 9735   | 10707   | 11318.5  | 10706.8  | **11652** |\n| Pong    | 14.6   | 20.9   | 21.0   | 21.0  | 21.0   | **22.1** |\n| Seaquest   | 42054.7 | 26434  | 30140   | 19401  | 29383.3  | **38061**  |\n| Tennis     | -8.3 | 23.1   | 23.6  | 5.8   | 22.6   | **27.4**  |\n| Venture  | 1187.5  | 1520   | 1318  | 1107  | 1112   | **1429**  |\n\n**Q5**: *Moreover, I just noticed a paper... Could you clarify the differences .........*\n\n**A5**: Thanks for your comments. We make the clarification as below:\n\n(i) **The paper  was not yet publicly available (this paper was accepted in July 1st) when our paper was submitted (our work was submitted in May 15th).** Hence, this paper was not cited in our work. \n\n(ii) **The differences** of our method from the paper DSAC-D **are obvious**:  Our method (VDRL) establishes **a theoretical foundation** between the variational bound objective (VBO) of diffusion model and value distribution learning objective (VDLO). **Building upon this theoretical foundation, we propose our method with optimal process and sample selection**, which distinguishes from DSAC-D you said.\n\n**Q6**: *I noticed that the performance ..................*\n\n**A6**: Thanks for your comments. We clarify this question from the following three aspects:\n\n(i) The difference of performance between DACER [10] and our method comes from **different implementation settings**. As shown in Figure 1 of DACER, **the x-axis is not the online interaction steps as in our work (i.e., training step) but the training iteration with 20 interaction steps**. Thus, to perform fairly comparison between DACER and VDRL, we need to multiply all the coordinates on the x-axis by 20. With this operation, you will obtain results similar to ours on DACER. \n\n(ii) **Our setting strictly follows the existing works**: SAC, TD3, DIPO, QSM, QVPO, etc. Our experimental results on all other baselines including SAC are also consistent with these works. \n\n(iii) Following your suggestion, we run experiments on three tasks of MuJoCo benchmark by using the official code of DACER (JAX), as shown in Table 5. In particular, to perform fair comparison, we adjust one training iteration with 1 online interaction step, which is same as the implementation setting of most existing works (i.e., SAC, TD3, DIPO, QVPO).**The performance of DACER (JAX) is similar as the experimental results of DACER (Pytorch) in the Table 1 of the original manuscript**. \n\nTable 5 The performance of DACER on three tasks of MuJoCo benchmark, which shows that the best mean returns and standard deviations over 800K training steps.\n|    | Ant-v3  | Halfcheetah-v3   | Walker-v3  |\n|:-----:|:------:|:-----:|:-------:|\n| DACER (JAX)    | 5493.57 ± 96.31 | 11428.16 ± 131.68 | 4062.09 ± 56.22 |\n| DACER (Pytorch)| 5506.40 ± 99.67 | 11250.96 ± 119.48 | 4053.14 ± 62.19 |\n\n[1] Shutong Ding, et al. Diffusion-based reinforcement learning via q-weighted variational policy optimization. NIPS, 2024."}]

Author rebuttal:

Meta-review: The reviewers appreciated the novelty of the paper, but converged on a positive view of this paper only after the authors provided substantial additional results that were not included in the submitted paper. Since it would not difficult to incorporate these results into a final paper, I am leaning positive on this.
Decision: Accept (poster)