Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization
Authors: Shutong Ding
Ke Hu
Zhenhao Zhang
Kan Ren
Weinan Zhang
Jingyi Yu
Jingya Wang
Ye Shi
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion Model
Reinforcement Learning
Q-weighted Variational Policy Optimization
Primary area: reinforcement_learning
TL;DR: We propose a novel diffusion-based online RL algorithm, conducting policy optimization with Q-weighted variational loss and diffusion entropy regularization to exploit the expressiveness and exploration capability of diffusion policy.

Abstract: Diffusion models have garnered widespread attention in Reinforcement Learning (RL) for their powerful expressiveness and multimodality. It has been verified that utilizing diffusion policies can significantly improve the performance of RL algorithms in continuous control tasks by overcoming the limitations of unimodal policies, such as Gaussian policies. Furthermore, the multimodality of diffusion policies also shows the potential of providing the agent with enhanced exploration capabilities. However, existing works mainly focus on applying diffusion policies in offline RL, while their incorporation into online RL has been less investigated. The diffusion model's training objective, known as the variational lower bound, cannot be applied directly in online RL due to the unavailability of 'good' samples (actions). To harmonize the diffusion model with online RL, we propose a novel model-free diffusion-based online RL algorithm named Q-weighted Variational Policy Optimization (QVPO). Specifically, we introduce the Q-weighted variational loss and its approximate implementation in practice. Notably, this loss is shown to be a tight lower bound of the policy objective. To further enhance the exploration capability of the diffusion policy, we design a special entropy regularization term. Unlike Gaussian policies, the log-likelihood in diffusion policies is inaccessible; thus this entropy term is nontrivial. Moreover, to reduce the large variance of diffusion policies, we also develop an efficient behavior policy through action selection. This can further improve its sample efficiency during online interaction. Consequently, the QVPO algorithm leverages the exploration capabilities and multimodality of diffusion policies, preventing the RL agent from converging to a sub-optimal policy. To verify the effectiveness of QVPO, we conduct comprehensive experiments on MuJoCo continuous control benchmarks. The final results demonstrate that QVPO achieves state-of-the-art performance in terms of both cumulative reward and sample efficiency.

Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization
Shutong Ding 1.3
Ke Hu'
Zhenhao Zhang
Kan Ren 1.3
Weinan Zhang?
Jingyi Yu!.3
Jingya Wang
Ye S
ShanghaiTTch University Shanghai Jiao Tong University Moo Key Laboratory of Intelligent Perception and Human Machine Collaboration
(dingsht, v-huke, v-zhangzhh1, renkan) edu. cn (yujingyi, wangjingya, shiye)@shanghaiteen tech, edu, on wnzhangās tu. wat..edu.com cn
Abstract
Diffusion models have garnered widespread attention in Reinforcement Learning (RL) for their powerful expressiveness and multimodality. It has been verified that utilizing diffusion policies can significantly improve the performance RL algorithms in continuous control tasks by overcoming the limitations of unimodal policies, such as Gaussian policies. Furthermore, the multimodality of diffusion policies also shows the potential of providing the agent with enhanced exploration capabilities. However, existing works mainly focus on applying diffusion policies in offline RI, while their incorporation into online RI. has been less investigated. The diffusion model's 's training objective, known as the variational lower bound, cannot be applied directly in online RL due to the unavailability of "good' samples (actions). To harmonize the diffusion model with online RL, we propose a novel model-free diffusion-based online RL algorithm named Q-weighted Variational Policy Optimization (QVPO). Specifically, we introduce the Q-weighted variational loss and its approximate implementation in practice. Notably, this loss shown be tight lower bound of the policy objective. To further enhance the exploration capability of the diffusion policy, we design a special entropy regularization term. Unlike Gaussian policies, the lig-likelihood in diffusion policies is inaccessible; thus this entropy term nontrivial. Moreover, to reduce the large variance diffusion policies, we also develop an efficient behavior policy through action selection. This can further improve its sample efficiency during online interac- tion. Consequently, the QVPO algorithm leverages the exploration capabilities and multimodality of diffusion policies, preventing the RL agent from converging to sub-optimal policy. To verify the effectiveness of QVPO, we conduct compre- hensive experiments on MuloCo continuous control benchmarks. The final results demonstrate that QVPO achieves state-of-the-ar performance in terms of both cumulative reward and sample efficiency. Our official implementation is released in /tttts:://gthiii com com./vadx2019/yypo)
Introduction
Recent years have witnessed significant success in the application of diffusion policy in imitation learning 6 22
27 32 and offline reinforcement learning 44 16 5 26 imitation
"Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

learning, diffusion models [3 35| 7 are often employed to capture the intricate distributions found within expert datasets. Furthermore, in offline RL, certain works replace unimodal policies with diffusion models to enrich the diversity in sampled actions or decision sequences. This success can be largely attributed to the robust expressiveness and multimodality inherent in diffusion models [13]
Such methods have demonstrated their effectiveness in practical scenarios like robotic navigation 136
43), robot arm manipulation 6 [7 33 23 dexterous hand and legged robot locomotion control
While diffusion policies have been extensively explored, their application in online RL has received relatively less attention. In online RL, value estimation varies with policy changes 45), posing a significant challenge. While diffusion models can effectively capture data distribution, they cannot directly improve the policy due to their training objectives [42.. Consequently, integrating diffusion models into traditional online RL framework [34] [0] 9 38 is challenging. Some works, such as
DIPO propose leveraging Q-function gradients to update actions for higher rewards, followed by employing diffusion models to fit the updated action distribution. However, relying on gradient updates constrains the algorithm's is exploration capability. QSM [3] directly aligns the score and the gradient of the Q-function under the perspective of score-matching. However, since the gradient of the Q-network is inaccurate, QSM has a doubled approximation error from the alignment process, which prevents the policy from converging to optimality.
To address these issues, we propose a novel model-free online RL algorithm called Q-weighted Variational Policy Optimization (QVPO). The core idea behind QVPO is straightforward yet effective. By revisiting the VLO objective of diffusion models and the policy objective of online RL, we discovered that the Variation LOwer Bound (VI.O) objective, with appropriate weights for state-action pairs, can form tight lower bound of the policy objective under certain conditions. This new objective, termed Q- weighted VLO loss, is complemented by Q-weight transformation functions, allowing it to be applied to general RL tasks. Additionally, we add an entropy term to the diffusion policy loss to enhance exploration. Calculating the exact entropy is intractable due to the inaccessibilityy of the probability density function (PDF) of diffusion policies. Therefore, we design an entropy regularization term from another perspective. Moreover, while diffusion policies offer advantages like multimodality, they also introduce large policy variance, leading to inefficient interaction with environments. To address this, we develop efficient behavior policy through action selection to improve sample efficiency. With QVPO, diffusion policies can fully leverage their exploration capability and multimodality in the online RL. paradigm. QVPO achieves state-of-the-art performance in terms of both cumulative reward and sample efficiency compared to traditional online RL. methods and existing diffusion- fffdee RL methods in MuJoCo locomotion tasks (39]
Our contributions are summarized as follows:
1) weiihtte VLO Loss. By revisiting the VLO loss of diffusion models and policy loss in online RL, we propose the Q-weighted VLO loss, which is the core component of the proposed QVPO method. We further extend its application to general RL tasks via equivalent Q-weight transformation functions. Additionally, we theoretically prove that the Q-weighted VLO loss a tight lower bound of the policy objective in online RI.
2) Diffusion Entropy Regularization & Efficient Behavior Policy. We find that the exploration ability of diffusion policies declines with limited diffusion steps. To address this, we apply a special entropy regularization term to the policy loss, which is nontrivial for diffusion policies. Additionally, to cope with the large policy variance of the diffusion model, we develop an efficient behavior policy through action selection improve sample efficiency.
3) State-of-the-art Performance. We verified the effectiveness of QVPO on MuJoCo locomotion benchmarks. Experimental results indicate that QVPO achieves tfate-ff-the-art performance in terms of sample efficiency and episodic reward compared to both previous traditional and diffusion-basee online RL algorithms.
2 Related Works
In this section, will review the existing works that utilize diffusion models in decision-making tasks and generally divide them into four categories according to different applications.
Diffusion Policies for Imitation Learning, Imitation learning is framework for learning a behavior policy from expert datasets without explicit reward. Diffusion models can effectively fit the distribu- tion of given datasets due to their powerful expressiveness. Some research works such as Diffusion
2

---

Policy (6,, Crossway Diffusion (22), AVDC [8] and (27 exemplify this by generating robot action sequences via diffusion based policy conditioned on visuomotor observation. Considering the reward is sparse or inaccessible in the expert dataset, BESO 32 leverages the diffusion model in the domain of goal-conditioned imitation learning to learn goal-specified policy.
Diffusion Planners. Planning in RL involves using the dynamic function to inform decision-makin over a long horizon. Diffusion planners reduce compounding errors by directly constructing the complete trajectory rather than relying on one-step transitions. Diffuser [15] arranges state-action sequences into a structured wwo-dimensional array and samples the trajectories based on reward guidance. In subsequent work, Decision Diffuser [2] generates state sequences with classifier free guidance and then uses inverse dynamic function to derive actions to alleviate the distribution shift problem. Latent Diffuser [20) generates sequences in learned latent space and then reconstructs the original trajectories. A bunch of works such as 21 3 g leverage hierarchical structures to enhance learning efficiency and maaiiign-making capabilities by decomposing complex tasks into a hierarchy of sub-tasks.
Diffusion Policies for Offline RL. Unlike most imitation learning problems without reward and assuming an optimal expert provide the data, offline RL faces the challenge of learning optimal policy from suboptimal offline datasets [19]. Diffusion-QL. (40) integrates the diffusion model with the conventional Q-Jearning framework. However, the training of Diffusion-QU -LL is unstable in the out-of-distribure (OOD) state region, SRDP [1] reconstructs the state to alleviate the distribution shift caused by OOD. EDP [16 reduces the computation cost during diffusion inference by applying Tweedie's s formula to predict actions. CEP (24] draws samples by diffusion sampling with exact guidance defined by the energy function based on contrastive learning. CPQL [5] uses a consistency
model to represent the policy. Some works [12 4 11 utilize diffusion models to construct the
optimal policy by weighted regression 229 28
Diffusion Policies for Online RL. Online RL faces more challenging problem in that it requires algorithms to learn the policy and make decisions in real-time interaction with the environment. DIPO 42 is the first to employ diffusion policies in online RL and proposes novel policy im- provement method. During diffusion policy updates, each action in the reply buffer is updated via action gradient to receive higher rewards. However, the action of gradient updating may deviate from the behavior policy, and the paradigm of gradient updating limits the algorithm's 's ability of global exploration and raises computational costs. QSM [31 introduces new update role for diffusion policy by aligning the score of the diffusion model with the gradient of the Q-function to increase the Q-value of state-action pairs. However, it overlooks the impact of inaccuracies in the value function gradient and biases introduced during the alignment process when updating the policy. This can lead to the policy being influenced by suboptimal data within the buffer. Compared with these existing diffusion-basse online RL algorithms, QVPO performs policy optimization with the Q-weighted VLO loss which can be theoretically proven to be the tight lower bound of the policy objective of online RL. This means QVPO does not incur additional errors in policy optimization. Besides, diffusion policy entropy regularization and efficient behavior policy via action selection techniques are provided to further enhance the performance of the diffusion policy in online RL.
3 Preliminaries
3.1 Reinforcement Learning
For reinforcement learning (RL), an MDP is defined as A. r, Po, 7), where S is the state space, A is the action space, p: S x S x A + oo) is the transition probability function of the next state 8(41 given the current state 8t and the action ag r S x A- + [Fmin= S function, Po: : S + [0, oc) is the distribution of the initial state so and 7 E [o, 1] is the discount factor for value estimation. In an MDP, the process starts from an initial state so sampled from po and then samples actions a, from the policy (als) S x A- + [0, oo) given the state st. Here (So, 0p, $1, ... denotes this kind of trajectory and 7 ~ π denotes the distribution of trajectory 7 given the policy π (a s). The state-value function Vz= (s) E Ex= r're|su = s) represents the expected reward that the agent can obtain under the policy in the state s. Compared to the state-value function, the action-state function is Qr a) = E provided as input. Further, the advantage function refers to the extra benefit of taking a specific action relative to taking the average action at a given state which is defined as the following: Ar (8, a) = Qx (x, a) - V, (s). The goal of RL is to learn the policy that maximizes the discounted expected
T'nin mmax] is the bounded reward
(r = a, 8o = s) where action ag is
3

---

cumulative reward, defined as [TT = BrNN Ei=o r'r.). which can be optimized via performing multiple steps of policy improvement T arg max, Br-r, [Q (s, a) log( s)]].
3.2 Denoising Diffusion Probabilistic Models
Denoising diffusion probabilisticc models (DDPM) 13 are powerful latent variable generative models that transform any data distribution into simple Gaussian distribution by adding noise (forward process) and then denoise it using neural networks (reverse process). Given a dataset {x1 I for x ~ ((x)), the forward process of DDPM transforms the distribution g(2a) into tractable prior Gaussian distribution by incorporating Gaussian noise in 7 steps with the transitions:
(xxx xt-1) == (x v - B,1),
where 3r is the variance schedule. Using the Markov chain property, can obtain the distribution of De conditioned on Zo:
(x4 xo) = [xxx; Voixa (1 TT)
(2)
where az 1 Bg and O₂ = I -o ag. Eventually, when T - oo, 2T converges to isotropic Gaussian distribution. In the reverse process, DDPM first generates noise from the prior distribution p(xx)= = (Nx;;; 0, and then gradually denoises it by learning parameterized transitions px x²) = - 1oxx t), [x,,,, t)) to fit g(; Xo),, where 0 denotes the learnable parameters.
The training objective of DDPM to maximize the Variational LOwer Bound (VLO) defined as log (x(o)))) 9(x..r (x6)
LVLO Bx(x)))
Finally, with c ~ (N( I), the loss in DDPM takes the form of:
E EI
[[|e: - -e E + vI - a,er, e)]
4 Q-weighted Variational Policy Optimization
In this section, we will first revisit the VLO loss [3] of the diffusion model and the policy objective of online RL (37). Then, we derive the Q-weighted VL.O loss, which is the principal component of the proposed Q Q-weigtted Variational Policy Optimization (QVPO). However, the Q-weighted VLO loss cannot be directly applied in general RL tasks. To address this issue, we also provide equivalent Q-weight transformation functions. Since the derived Q-weighted VLO loss is the tight lower bound of the policy loss in online RL, QVPO can effectively avoid the drawbacks of existing online RL.
methods for diffusion policies 42 31
mentioned in Section 2
QVPO also involves practical techniques for the underlying issues of optimizing diffusion policies. Firstly, to further enhance the exploration ability of diffusion policy, we develop a special entropy regularization. This not trivial for diffusion policy since the log probability of the state-action pair is not available. Besides, we devise an efficient behavior policy via action selection for the diffusion model avoid the large policy variance of diffusion, which results in sample inefficiency. With these practical techniques, diffusion policy can achieve better performance with fewer online interactions with the environments. Finally, the training procedure of QVPO is shown in Figure T
4.1 Q-weighted Variational Objective for Diffusion Policy
As mentioned in DIPO [42), optimizing diffusion policies in the online RL paradigm is nontrivial, which principally results from two reasons. In one aspect, if we directly apply the deterministic policy gradient to diffusion policies like Diffusion-QL. [40] the backpropagation chain through the denoising procedure of the diffusion model becomes quite long. This leads to high computational costs and instability during training, which severely restricts the performance of diffusion policies in online RL. In another aspect, samples from the optimal policy are required when directly using the variational bound objective to train diffusion policies. However, these optimal samples are typically unavailable in online RL.
In this context, we revisited the VLO objective of diffusion models and the RL policy objective. To our surprise, we discovered that by adding the appropriate weights the VLO objective, it becomes a tight lower bound of the RL policy objective under certain conditions.
Theorem 1. (Lower Bound of RL Policy Objective) If ([{(sa)) 2 0 for any state-action pair (8, a). the Q Q-weittted variational bound objective of diffusion policy
To (ao:T s)
Eg Ex.a--lss (...a_ppsll...)...a
8.0). (a|s)
Q(s, ao) (aa)) (sgranga))))))
log
g (a 8, ao)

---

Diffusion policy
Rx-1 8,0c0
Q(s,a)
V1 a)
or
D4
04-1
5,44 1)
Mutippe action samples
Assign weights actions
Action uniform distrbution
Updating diffusion policy
Q-weighted VLO Loss:
ace, (o (ayls a) I|e co (Voia vie (11
Diffusion Entropy Regularization:
2a (one(s)
| ca (Vāia vI- 8,
Visualizatin locomation
The training pipeline of QVPO. In each training epoch, QVPO first utilizes the diffusion policy to generate multiple action samples for every state. Then, these action samples will be selected and endowed with different weights according to the Q value given by the value network. Besides, action samples from uniform distribution are also created for the diffusion entropy regularization term. With these action samples and weights, we can finally optimize the diffusion policy via the combined objective of Q-weighted VLO loss and diffusion entropy regularization term.
Figure 1:
is the tight lower bound of the objective of RL policy
E Eesarrp(e')eeo (s.pple(ss.)),..omm mx(a)e) [Q(s, a) log(To (a(s)))
and the equality holds when the policy converges.
The proof can be referred to in the supplementary materials. According to this theorem, we can derive the Q-weighted variational bound loss as (4) which can applied diffusion policy optimization.
3
(()) A Eg (ssanoaa(asss...
((s a). ||e EE (Vara + v1 a,e, s, +)
(4)
207a, (1 ā.)
of 13 removing the coefficient (1-a) does not affect the training of diffusion.
According
Hence, the final Q Q-eigghted VLO loss is defined as
(0) 2 Es,a=m,(als)... [Q(s.a) a) |||c - (Vaia+ āic, tt]l
(5)
While we can now use (5) to optimize the diffusion policy, two issues remain to be resolved.
1) Negative Q value. In real-world decision- decision-making tasks, it is difficult to ensure that the returned reward is always non-negative, which means the Q value may be negative for some state-action pairs. Therefore, to apply QVPO in practical tasks, we must address situations where the Q(8, a) value is negative for certain states and actions.
2) High-quality Training Samples. According to (5), achieving significant policy improvement requires obtaining certain rare state-action samples with high Q values. However, this is significant challenge with limited interaction with the environment. This issue is also common in other online RL methods based on weighted policy training, such as RWR (30, and hinders their application to real-world tasks.
4.2 Equivalent Transformation for Q-weighted VLO Loss
To resolve these problems in training diffusion policies with the Q-weighted VLO loss, propose equivalent Q-weight transformation functions to convert the Q value into an equivalent positive Q weight and leverage the powerful data- sattaasytteesziing ability of the diffusion model to generate high-quality samples for training. The new equivalent Q-weighted VLO loss is defined in (6). We will introduce the detailed implementation of these solutions in the following contexts.
(())= Eg (ale).e.t [e a) || Eg (Vaga + VI VI-a,6 8, 1
5

---

Theorem 2. (Optimal Solution in One Policy Improvement Step) Considering the optimization problem in policy improvement step T arg max, E,,,, 1m, (s, a) (o s))] the optimal policy in given state is
[aals)((s.o) T oooo
Q(s,a) >
p(a 8) =
(7)
0,
otherwise
when there exists such that ((8,a) > O, and
1 N Σ 8(x (())-a),a as {a ((l)(ssa)) max a(s, T(a 00] :
p(a s)
(8)
when (s. S O for any E A. 8(x a) is the Dirac function, N is the cardinality of set {a Q(..a) max, Q(8,a), a) T(a
())> >0], xa 8) is the behavior policy, p(a 8) is the policy to [
be optimized and =))(s))> is the indicator function that judges whether (s, 0.
The proof can be referred
the supplementary materials. Hence, with Theorem we can obtain the probability density function (PDF) of the optimal policy in one policy improvement step and
[a]ss) as the weight to achieve equivalent optimized diffusion policy according to (6) However, the difficulty of judging whether any (ss, a) S 0 in a certain state & is still problem. Therefore, we first come up with an approximate solution "ycut" weight transformation function, which returns ((s, a) when ((s, a) 2 and small value > when Q(s, a) < O. Furthermore, the "qeut" weight transformation function just endows the best action with the weight mentioned above and discards the left action samples via setting them as zero.
further use p(ale)
In short, qeut weight transformation function takes the formulation of the above two situations into account and "cuts" them via small value > Notably, we remove the denominator I [Q ssQ(s, a)da in geut weight transformation function. This is because the denomina- tor f (a)) (r | ss)(s, a)da is somewhat like the Vs)), which denotes the "importance" of the state s. Therefore, removing this item can allow important states to obtain larger weights during training. which is confirmed in practice.
However, we find that the qeut weight transformation function does not work very well when Q values in most state-action pairs are less than zero. To address this issue, we propose another "gadv" weight transformation function, which is much simpler but performs better in our experiments.
Wey Wey A Wgade (g (s A(s,a), 0,
A(s,a) >
(9)
A(s,a) <0
where A(s, a) Q(s (s) ) is the advantage function. qadv weight transformation function replaces the Q(s, a) with the advantage A(s, a) avoid the second situation. Moreover, applying Q value and advantage in policy optimization has been proved equivalent and can also reduce the training variance in previous works like (25
is worth noting that, for qadv weight transformation function, we do not require an extra network to approximate Vss) calculate A(s, a). We can always sample certain number of actions for state s with diffusion policy and estimate V(s) using the average of their (s,, a). Besides, for qadv function, we also recommend just selecting the best action sample with the largest A value for each
state in policy optimization, which can further improve the quality of training samples and reduce the training cost.
Remark. Although the weighted VLO loss of QVPO looks similar to AWR 28 and EDP [6], they different intrinsically. With the Q-weighted VLO loss, QVPO trains the diffusion policy with the weighted sample from the current policy, while AWR or EDP trains the diffusion policy with the weighted sample from the policy in the offline dataset (replay buffer). These "on-policy" samples truly benefit the online training. Moreover, AWR and EDP utilize the exp weight function; but this function is too conservative for online RL.
4.3 Enhancing Policy Exploration via Diffusion Entropy Regularization
While diffusion policies can achieve impressive performance with the Q-weighted VLOO loss online RL, their powerful exploration capabilities have yet to be fully harnessed. As noted in Diffusion-QU

---

Diffusion Policy w/o entropy term
Training Procedure
Diffusion Policy w/ entropy term
Explorable area of diftusion policy
Unexplorable area
Target objective fundtion
Figure 2: A toy example continuous bandit to show the effect of diffusion entropy regularization term via the changes of the explorable area for diffusion policy with the training procedure. The contour lines indicate the reward function of continuous bandit, which is an arbitrarily selected function with 3 peaks.
40). the policy expressiveness of the diffusion model decreases with fewer diffusion steps. In experiments, we find that not only does policy expressiveness decline, but the exploration capability of the diffusion model also diminishes when the number of diffusion steps is reduced. However, is essential to limit the number of diffusion steps to avoid excessive training and evaluation costs in real-world applications. Therefore, studying how to enhance the exploration ability of the diffusion model with limited number of diffusion steps is necessary.
Adding extra entropy regularization term the policy loss appears be good solution, as has been validated for categorical policies discrete action spaces and Gaussian policies in continuous action spaces. However, estimating the entropy for diffusion policy is nontrivial due to the inaccessibility of the log-likelihood of action samples. Moreover, maximizing the entropy of policy can be viewed as narrowing the distance between the policy and maximum entropy distribution (i.e., uniform distribution) in some sense. Thus, by recalling the VLO loss of the diffusion model, the entropy of diffusion model can be increased with training samples from the uniform distribution. Based on this idea, we propose an entropy regularization term for diffusion policies as follows.
(0) 2 Eggawthee. Went (8) ||e - ( +
(10)
where (ent(s) = went 21=1 N between exploitation and exploration. Here, N represents the number of selected training samples from the diffusion policy for state s. When Went is large, the exploration ability of the diffusion policy is improved, whereas the exploitation ability is reduced. Figure 2is the experiment results on a continuous bandit toy example that clearly illustrates the effect of this entropy regularization term on the diffusion policy, and the conerete reward function of this bandit problem is R(x:) [-1. 0.65)T, m:)). where w; 1.5, 0.1, and μı (- [-0.65, 1.35]T and [-166,, 1.61]T respectively.
(s,a)) is the coefficient related to state s for balancing the trade-off
E 2 exp (x 1²) (x (m:))
4.4 Reducing Diffusion Policy Variance via Action Selection
Despite the diffusion model that allows the online RL agent seek better policies, also introduces a large policy variance. This results in inefficiency for online interactions of the behavior policy with environment. To address this issue, we propose an efficient behavior policy via action selection for diffusion model T (a s), which improves the sample efficiency. Specifically, it is motivated by the idea that the efficiency of action samples from behavior policy depends on their Q values. In other words, action samples from behavior policy with high Q values indicate that the following trajectory also has a large underlying reward and of great significance for training. Specifically, the efficient behavior policy T (a is defined as:
K (a (s)
argmax of (aI (als))
a(s,a).
Furthermore, while using the efficient behavior policy m (a 8) as behavior policy can improve sample efficiency, it is not recommended to apply the same action selection number K to the target
7

---

policy for calculating the TD target. This is due to the large policy variance of the diffusion policy, which can result in a serious overestimation of the target Q value. Hence, if we choose the action selection number K5 for the behavior policy, smaller action selection number K₁ < K₁ for the target policy often good choice in practice.
Table Comparison of QVPO and other online RL algorithms in evaluation results, (N/A indicates the algorithm does not work)
Invrirumments
PPO
SPO
TD3
SAC
DIPO
QSM
QVPO(*)
Hopper-v3
2212.80988 2154,3(4262)
52.5.5(8.5)
2996.61
0)
2154,7(998,,) 2)
(7..8.5(33.8)
Walker2d-v3
3751.5(609.1)
3321 11328.9)) 9)
3513.9(40.7)
468 1625.7(25.7)
1613.4((443.55
5191.8(60.2)
Ant-v3
2781.974.1)
2100.2(302.4
456..8669.5)
5665.9(54.7)
N/A
6625.1
HalfCheerah-w3
4773.5(53.4)
4008. 4006.2(246.8) 8)
10388.6680-4
10616.9(7288
9590.5(67.5)
3863.2(632.6)
113355661644
Humamoid-v33
713.7(85.9)
797.4(262.1)
5353.5(53.7)
5159.7(475.3)
4945.5(898.6)
4793.1(229.55
5306.6((4.55
5 Experiments
In this section, we first describe a practical implementation of QVPO, as shown in Algorithm 1 which is implemented in off-policy fashion. Then, we evaluate the proposed algorithm different decision-making tasks and with various hyper-parameters. With these experimental results, we aim answer three questions:
How does QVPO compare to previous popular online RL algorithms and existing diffusion- based online RL algorithms?
How does the entropy regularization term affect the performance of QVPO?
* Can the K-efficient behavior policy significantly improve the sample efficiency of QVPO?
Algorithm Q-weighted Variational Policy Optimization
Input:
Diffusion policy (a s), value network Qw a). replay buffer D, Ky- Kffffiicceent diffusion policy for behavior policy, Ke-effiecient diffusion policy for target policy, number of training samples Na from diffusion policy, number of training samples Ne from uniform distribution 24 (((a, a).
1: for in 1, ... . do
2: Sample the action using the diffusion policy #g Ka (a (s)).
3: Take the action o, in the environment and store the returned transition in D.
4: Sample a mini-batch B of transitions in D.
5:
Generate Na samples from (a s), and N, samples from (Q, () for each state in B.
6:
Endow the Na samples with weights (9).
7:
Select an action sample amaz with maximum weight among Na samples for training.
8:
Endow the Ne samples with the weight Went (s) = Went (s, amax)) )
9: Update the parameters of the diffusion policy using the summation of (6) and (TO).
Construct TD target as ye Ft + Quw m (a 1)+1)) for each (st, az, Ft, S141) in B.
11: Update the parameters of the value network using MSE loss.
end for
5.1
Comparative Evaluation
To demonstrate the effectiveness of our method, we compared QVPO with six other online model-free
RL algorithms: two off-policy algorithms (TD3 9 and SAC IO),, two on-policy algorithms (PPO [34] and SPO (3), and one advanced diffusion-based online RI. algorithm (DIPO [42].
QSM 31). These comparisons were conducted on five MuJoCo locomotion tasks [3). we plotted the learning curves of QVPO and the six other algorithms over five random runs, shown in Figure
3 The solid curve represents the average return, and the transparent shaded region represents the standard deviation. Each experiment was conducted over 1e6 training epochs. According to the learning curves, QVPO achieves state-of-the performance compared to the other six algorithms, and converges much faster than the other algorithms, further demonstrating its sample efficiency. Additionally, the practical implementation QVPO follows SAC in critic part, which utilizes doubled Q networks and only the minimum for policy and critic update.
Table I presents the evaluation results of QVPO and six other algorithms on the MuloCo locomotion tasks after 1e6 iterations, further confirming the superiority of QVPO. Notably, during the evaluation
8

---

Hopparor3
Walker
(C) Antiv33
2510
6000
2000
4230
E2 Some
2000
1550
2330
2000
6aoch
Rooch
F2eth
(D) HallCheatiawee
(3) Humanoidew3
6920
4380
2300
Spack
6pem
Figure 3: Learning Curves of different algorithms on Mujoco locomotion benchmarks across 5 runs. The x-axis is the number of training epochs. The y-axis the episodic reward. the plots smoothed with window of 5000.
stage, apply the efficient behavior policy with large action selection number K = 32 instead of the deterministic denoising procedure used in DIPO 42). As illustrated in [41], the high- high-probabiility region in each diffusion step is not the origin but Gaussian sphere. Hence, using the deterministic denoising procedure in the evaluation stage not a good choice. In that case, using the proposed efficient behavior policy during evaluation is more reasonable. More details related the experiments can be found in the supplementary materials.
5.2 Ablation Study and Parameter Analysis
To further examine the significance of each component in QVPO, we also conducted the ablation study and parameter analysis on the effects of the entropy regularization and the K K-efficient behavior policy. Here We use the experiment on Ant-v3 as an example since the final results are similar across different locomotion tasks.
5000
2000
Effect of Difffusion Entropy Regularization. As shown in Fig- ure without the diffusion entropy regularization loss, QVPO with 100 diffusion steps achieves much better performance than QVPO with 20 diffusion steps. However, when the entropy regularization loss is applied, QVPO with diffusion steps achieves close performance to QVPO with 100 diffusion steps, which verifies the effectiveness of the entropy regularization term.
Figure 4: Comparison between QVPO with and without the diffu- sion entropy regularization.
Action Selection number K for Efficient Behavior Policy, Figure the performance of QVPO with different ac- tion selection numbers for behavior policy K₁ and for target policy K,. It can be observed that QVPO with action selection numbers K₁ 4, K₁ 1 is superior to QVPO with action selection numbers Kg 1, K; = 1, while the training stability K; is better
2000
2000
1000
of QVPO with K2 4, K, than that of QVPO with K₂ = 4, K2 4. This indicates that ef- ficient behavior policy via action selection can improve sample efficiency and setting the action selection numbers K, < K₁ can help reduce the overestimation error of the target Q value. Besides, the experiment with K₁ 20, K₁ 2 shows that
and K2
Specch
Figure 5: Comparison of QVPO with different action selection num- bers for behavior policy Kú, and for target policy Kr.

---

too-high action selection number will lead to limited exploration. Therefore, setting K₁ = 4 can balance exploration and exploitation well.
Conclusion
In this paper, we proposed a novel diffusion baasudd online RL algorithm called QVPO that sufficiently exploits the expressiveness and multimodality of diffusion policy, QVPO leverages the Q-weighted VLO loss, a core component that serves as a tight lower bound of the policy objective online RL under certain conditions, facilitated by Q-weight transformation functions. Additionally, we designed a special entropy regularization term to enhance the exploration capabilities of diffusion policies and the efficient behavior policy to improve sample efficiency by reducing the behavior policy variance. Our comprehensive experiments on MuJoCo continuous control benchmarks demonstrate that QVPO achieves ofaee-of-the-ar performance in terms of both cumulative reward and sample efficiency, surpassing both traditional and existing diffusion- based online RL methods.
However, there are still several underlying challenges in applying diffusion policies to online RL that require further exploration. For instance, although we developed special entropy regularization loss to approximate the effect of maximizing entropy, it lacks the ability to adaptively adjust the entropy term and cannot incorporate the entropy of the diffusion policy into the TD target for soft policy iteration, as seen in SAC. Future work will focus on developing adaptive entropy adjustment mechanisms and integrating entropy into the TD target to enable soft policy iteration, which we believe will further enhance the performance of diffusion policies in online RL.
Acknowledgement
This work was supported by NSFC (No.622) 6230319,,,,, Shanghai Sailing Program (22YF1428800), Shanghai Local College Capacity Building Program (23010503 100), Shanghai A14S Initiative SHTA14S202404, Shanghai Frontiers Science Center of Human-centered Artificial Intelligence (ShangHAI), HPC Platform of ShanghaiTecch University, MoF. Key Laboratory of Intelligent Per- ception and Human-Maching Collaboration (ShanghaiTechh University) and Shanghai Engineering Research Center of Intelligent Vision and Imaging. We also thank Ming Zhou, Shanghai AI Labora- tory, for his discussion and suggestion in our proof of Theorem 4.1
References
[1] Suzan Ece Ada, Erhan Oztop, and Emre Ugur. Diffusion policies for out-of-distribution generalization in offline reinforcement learning. IEEE Robotics and Automation Letters, 966):31 April 2024.
[2] Anurag Ajay, Yilun Du Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for making? arXiv preprint arXiv:2211.15657. 2022.
[3] Chang Chen, Fei Deng, Kenji Kawaguchi, Caglar Gulcehre, and Sungjin Ahn. Simple hierarchi- cal planning with diffusion. arXiv preprint arXiv:2401.00 102644, 2024.
[4] Huayu Chen, Cheng Lu, Chengyang Ying. Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. arXiv preprint arXiv:2209.1454 2022.
[5] Yuhui Chen, Haoran Li, and Dongbin Zhao. Boosting continuous control with consistency policy. arXiv preprint arXiv:2310.06344 2023.
[6] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchtiel, and Shuran Song, Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint 0rX1v:2303.04137, 2023.
[7] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-877 87944, 2021.
[8] Zibin Dong, Jiane Hao, Yifu Yuan, Fei Ni, Yitian Wang, Pengyi Li, and Yan Zheng. Diffuser- Lite: Towards real-time diffusion planning, arXiv preprint 1rX1v:2401.15443, 2024.
10

---

[9] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596.
PMLR, 2018.
[10] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[11] Philippe Hansen-Este -Ettrcch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit g-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304. 10573, 2023.
[12] Longxiang He. Linrui Zhang, Junbo Tan, and Xueqian Wang. Diffeps: Diffusion model based constrained policy search for offline reinforcement learning. arXiv preprint arXiv:2310.05333, 2023.
[13] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusiin probabilistic models. Advances in neural information processing systems, 33:684006851,, 2020.
[14] Xiaoyu Huang, Yufeng Chi, Ruofeng Wang, Zhongyu Li, Xue Bin Peng, Sophia Shao, Borivoje Nikolic, and Koushil Sreenath. Diffuseloco: Real-time legged locomotion control with diffusion from offline datasets, 2024.
[15] Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.0999) 2022.
[16] Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[17] Tsung- Wei Ke, Nikolaos Gkanatsios, and Katerina Fragkiadaki. 3d diffuser actor: Policy diffusion with 3d scene representations, 2024.
[18) Po-Chen Ko, Jiayuan Mao, Yilun Du, Shao-Hua Sun, and Joshua B. Tenenbaum. Learning to act from actionless videos through dense correspondences, 2023.
[19] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint 20XXi:00005.06443 2020.
[20] Wenhao Li. Efficient planning with latent diffusion. arXiv preprint arXiv:2310.0000 2023.
[21] Wenhao Li, Xiangfeng Wang, Bo Jin, and Hongyuan Zha. Hierarchical diffusion for offline decision making. In International Conference on Machine Learning, pages 20035-20064.
PMLR, 2023.
[22] Xiang Li, Varun Belagali, Jinghuan Shang, and Michael S Ryoo, Crossway Diffusion: Improving diffusion-based visuomotor policy via self-supervised learning. arXiv preprini arXiv:2307.01843 2023.
[23] Toru Lin, Yu Zhang, Qiyang Li, Haozhi Qi, Brent Yi, Sergey Levine, and Jitendra Malik. Learning visuotactile skills with two multifingered hands, 2024.
[24] Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. In International Conference on Machine Learning, pages
PMLR, 2023.
[25] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce- ment learning. In International conference on machine learning, pages 1928-1937. PMLR, 2016.
[26] Fei Ni, Jianye Hao, Yao Mu, Yifu Yuan, Yan Zheng, Bin Wang, and Zhixuan Liang. Metadiffuser: Diffusion model conditional planner for offline meta-rl. In International Conference Machine Learning, pages 26087-26118 PMLR, 2023.
[27] Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcareel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, and Sam Devlin. Imitating human behaviour with diffusion models, 2023.
(28) Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off ooolpy reinforcement learning. arXiv preprint
2019.
11

---

[29] Jan Peters, Katharina Mulling, and Yasemin Altun. Relative entropy policy search. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 24, pages 1607-1612. 2010.
[30] Jan Peters and Stefan Schaal. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning, pages 745-750, 2007.
[31] Michael Psenka, Alejandro Escontrela, Pieter Abbeel, and Yi Ma. Learning a diffusion model policy from rewards via q-score matching. arXiv preprint arXiv:2312.11752, 2023.
[32] Moritz Reuss, Maximilian Li, Xiaogang Jia, and Rudolf Lioutikov. Goal-conditioned imitation
learning using score-based diffusion policies. arXiv preprint arXiv:2304.025.3 2023.
[33] Paul Maria Scheikl, Nicolas Schreiber, Christoph Haas, Niklas Freymuth, Gerhard Neumann, Rudolf Lioutikov. and Franziska Mathis-Ulliich. Movement primitive diffusion: Learning gentle robotic manipulation of deformable objects, 2023.
[34] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1XXiv:/77.0665. 2017.
[35] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and
Ben Poole. Score-hased generative modeling through stochastic differential equations. arXiv preprint urXiv:2011.13455 2020.
[36] Ajay Sridhar, Dhruv Shah, Catherine Glossop. and Sergey Levine. Nomad: Goal masked diffusion policies for navigation and exploration, 2023.
[37] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
(38) Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
[39] Emanuel Todorov, Tom Erez, and Yuval Tassa Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012.
[40] Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2022.
[41] Lingxiao Yang. Shutong Ding, Yifan Cai, Jingyi Yu, Jingya Wang, and Ye Shi. Guidance with spherical gaussian constraint for conditional diffusion. arXiv preprint arXiv:2402.033: 0232002,020,, 2024.
[42] Long Yang, Zhixiong Huang, Fenghao Lei, Yucun Zhong, Yiming Yang, Cong Fang, Shiting Wen, Binbin Zhou, and Zhouchen Lin. Policy representation via diffusion probability model for reinforcement learning, arXiv preprint S.Ki:22305.13122 2023.
[43] Gengyu Zhang, Hao Tang, and Yan Yan. Versatile navigation under partial observability via value-guided diffusion policy, 2024.
[44] Zhengbang Zhu, Minghuan Liu, Liyuan Mao, Bingyi Kang, Minkai Xu, Yong Yu, Stefano Ermon, and Weinan Zhang. Madiff: Offline multi-agent learning with diffusion models, 2023.
[45] Zhengbang Zhu, Hanye Zhao, Haoran He, Yichao Zhong, Shenyu Zhang, Yong Yu, and Weinan Zhang. Diffusion models for reinforcement learning: A survey. arXiv preprint arXiv:2311.01223, 2023.
12

---

A Proofs
A.1 Proof of Theorem
Theorem 1. (Lower Bound of RL Policy Objective) If (s, a) the Q-weighted variational bound objective of diffusion policy
2 0 for any state-action pair (s, a),
log T (ao:T s)
Be.arppss (a (ale) Q(s, ao) (aarrn((a)))ss
g (AIr 8, 5,)
the tight lower bound of the objective of RL policy
E bssa [Q(s,
and equality holds when the policy converges.
Proof. The policy objective of online RI. is
max E pe.. [Q(s, a)) (ag(m)()))))],
0
where (a s) indicates the behavior policy in state s, no s) is the policy to be optimized, and Q(s, a) represents the Q value of the state-action pair (s, a). Then, if Ta(a 8) is a diffusion policy, the inner-term Q(s, a) log (To(a)))) equals to
(s,, un) ) rrgforr log
To (ao:r 8) (#I.T 8, 8,)
+Q(s, uo) Dez (q (air) 8, ao) || (aurr 88 (o))
According the Jensen's 's Inequality (convexity of log) and Q(s a) the second KL divergence term
(ss, ao) DEL (4 (air 8, 00)
TT (a1:T 8, (p))
log (a1.T
8, 4n)
((s, ao) Bo) g(autsaa)))
(ate
(0))
log To (a
88 8,)00)
(s, 0o) Bo0
(01.T
8,00)
T& (a1.7
8, 5,)
2 ( ao) log (oa.rg((ssz)1
8, 8,))
((s, ao) log(1)
0.
Obviously, the equality holds when my (a1:7 s.) q (a1:3) s, s,).). Finally, the Q-weighted variational bound objective
To (AO.T s) E Essnprptts/ QQ(s, an En log q (aiT s, ao)
the tight lower bound of the RL policy objective
E Eg (.))) [Q(s, a) log(TTo (als)))
13

---

A.2
Deriviation of Q-weighted VLO Loss
For each state s, we have
log 9 (a1. ao)
To (ao:T)
Il 9 (at (4-1)
log
Te (ar)
[[ To (a ar)
Il (as a1-1)
log TT (aT)
log
II Ty (a,-1 ax)
T
log ① (a+ at-1)
log To (ar)
+
(ac-1 az) 1=1
T
log (a+ (x-1)
q { an)
log To (ar)
+
+ log
(at-1 (t))
na (ao [a)
1=
9 (
ae. (c,00) (a+ ao)
(a) ao)
log To (aT)
+
log
+ log
T8 (ac-1
(00) (at-1 aa)
Tg (ao a1)
1=2
T
(
at, ao)
q (0c-1 (00) ) loog { ap) Te (ao a1)
9 (as [o))
log T (ar)
log
TT (ac-1 (z))
T
log 9 (a+-1
at, (0)
loog q q(at | ao)
(ac-1 aao) )
+ log ( (a)
log To (ar)
+
(ac-1 ae)
(ao a1)
t=2
T
2 (a
04, 0n)
g (ar an)
( ao)
log To (ar)
+
log
+ log
+ loog
To (a a1))
g (a1 an)
To (ap 01
1=
(a,-)
a2 ag)
E Eag reqaurrss
log (aT) +
Σ
log
+ lolo (ar an)
T (at-1 (e))
1=2
logg (a1 ao) logg (g)
loog n (ap a1)
log g (ur ao)
T
q (a--1
az, an)a0) log no (ao aa))
log
mo (AT)
To (a ax)
T
DKL (ar an) || (ar)) + Σ
DKL (q (ar-1
ac || (a.-1 a.) - log to (as aa1)
1=2
82
||e Eg (Vārau + vI -i--aee s, s,e, t)]1?]
= Ban.e 207a: (1 a.)
Thus, the Q-weighted VLO loss can be derived as
log (ao.r ()
max E, (.
Q(s, on) (EL
( s, ao)
log (aiT 8, 80
min E. (a a0) Besyao)eerrrgaaarlee
no (a0:T 8)
B 207a (1 (1-
min EE (a
a(s,a) ||e- Eo (Vāna + vI s,e,s 12)
A.3 Proof of Theorem 2
Theorem 2. (Optimal Solution in One Policy Improvement Step) Considering the optimization
problem in policy improvement step Tk-1 arg max, Eyroy (2 (s, a) log(z)), the optimal policy
14

---

in given state is
m[als)Q(s.a)
Q(s,a) 0
p(a 8) =
(1I)
0,
otherwise
when there exists a such that ()) > O, and
N
p(a 8) = N
1
Σ
8(x az), E fa Q(s,a) max m
(12)
when ((. < 0 for any E A. 8(z - a) is the Dirac function, N is the cardinality of set {a ((s,a) = max, Q(s,a), T( s) is the behavior policy, p(a s) the policy to be optimized and (()0))0 is the indicator function that judges whether Q(s, a) O.
Proof. Consider the optimization problem of the policy improvement step in a certain state s:
/ Taa 8)Q a) log p(a s)da
min p(a(s)
subject to
p(a s)da =
p(a 8) 0, A
where p(u s) a probability density function defined on A that represents the policy to optimized in the state
Here, we can derive the solution to this optimization problem according to KKT conditions by classifying it into three cases.
((s, a) > 0 for any action A. In this case, the optimization problem is convex, and can use KKT conditions directly to obtain its optimal solution as below:
. <a s))(s, a)
x v(a (8)
p(a!
f p(a s)da =1 v(a 8) 2
(13)
v(a s)p(a s) 0, A
Then, consider
p(als) <aa s)Q(s, x-a(a(s)
Since Q(s,a) > and p(als) 2 the denominator
A >- 8) 2 0.
Hence,
0, v(a 8) 0,
which means
aTa s)Q(s,a) p(a 8) J J s)Q(s.a)da
(ss, S for any action a E A.. When Q(s, a) & 0 for any action a E A, it can be found that the objective will be -oo if there exists action such that p(a|s) 0 and π ((a s) < O. In that case, only considering the original optimization problem not meaningful. Hence, we can further consider the optimality of f Q(s, a)p(als)da. Finally, we can drive the optimal policy pdf is the summation of the Dirac function
1 Σ 5(z= a:), ai E {a Q(s,a) max a(s,a) (a s) 0) N
p(a 8)
where N is the cardinality of set {a Q(s a) max. a(s.a),T(0 m(a })).
15

---

Q(s,a) > 0 for some actions and Q(s, a) I 0 for the left actions a. Combining the above situations, the optimal policy pdf is
((a,a)
Q(s, > otherwise
p(a s)
)
J o
Finally, we can summarize the optimal policy in a certain state as
a[als)Q(saa
Q(s,a)>0 0
p(a 8) -
(14)
0,
otherwise
when there exists such that Q(s, 0, and
N
p(a 8) N
Σ
8(x a.), a E fa Q(s,a) - max x
((a 10)>
(15)
when ((s, a) < 0 for any E A.
A.4 Proof for Convergence of QVPO
Assume the new diffusion policy after one QVPO iteration can be approximately expressed as
(n s) = (1 - Pdlata (s) Am (s, a" (s).""")))))))")))))))))))))(" Tota(a s) + Pdata (s) (s, a an) (a SN((a)ssc))
where a* is the action that can maximize Quace (s, a) the state 8, Va s, the neigh- borhood of a* with small radius e, Sx,(a is the area of this neighborhood, and paata (s) the sampling distribution the state. This assumption is straightforward: the training sample's generation probability in the diffusion model will be improved and the improved probability is proportional to its weight.
Now consider the improvement of the RI. objective:
I (Tnew) I (TTia) = Barop [Vine (8) Va
Earpp Bansnew (a|s) 12 Nnw (s, VasLa (8)
Baxpa [Borsnee (els) 1Q Rnow (8, Q rota []]]
Exsopo
E. Bowssew (a|s) (Qroa (8, a)] ViLnn (8)
= -E
(Bantaem 12 1new (s, a) Q (8, a)]] + Beparpo (als) (m a)]
The first term here can be further expanded according to the Bellman equation:
Es-ipo [Ea (als) (s, a) Qu a)] = >/Bandt
new Vanew (s) Vasua (8)]
where d_ di denotes the probability distribution of state in time step t = 1 with policy Tnew- Repeating the above operation, we will obtain:
I [(Tew) - I [(Tota)
o E. and nnw Tnew [Anoi (s, a)]
1=0
7 Exndenww [Baxsncw (sme a)]]
E Exwdenew
(1 Panta (s) (8, D (s,
+ Pdata (s)A3 (s, a² (eE
SN(a)
1
Eand Pduato (s, an)n InEN(ar(an)(a))
Sw(a-)aie)
16

---

B Hyperparameters
All of our experiments are implemented on a GPU of NVIDIA GeForce RTX 4090 with 24GB and CPU of Intel Xeon w5-3435X. The implementation of SAC, TD3, PPO, SPO, and DIPO is based on http://ps.sttttc...........o.tta
com http//uujjjjmmm/a TD3
and https: gi thub. com http://eimaaaaiiiitttttr//rrrrr/ rescctccii which are official code library. Table 2 and 3 present the papper-rrameterss used in our experiments. Notably, we find that the performance
https: / github........ com httphhhwww..hoooooooooooooooaoaa
of DIPO with 20 diffusion steps is worse. Thus, different from diffusion steps QVPO, 100
diffusion steps are set for DIPO, which are also recommended in the original paper 42
Table 2: Hyper-parameters used in the experiments.
Hyperparameters
QVPO
DIPO
SAC
TD3
SPO
PPO
No. of hidden layers
2
2
2
No. of hidden nodes
256
256
256
256
64
256
Activation
mish
mish
relu
relu
tanh
tanh
Batch size
256
256
256
256
256
256
Discount for reward y
0,99
0.99
0,99
0,99
0,99
Target smoothing coefficient 7
0.005
0.005
0.005
0.005
0.0
0.005
Learning rate for actor
3 x 10
3 X 10-1
3 X 10
3 10
3 10-4
7 x 10-1
Learning for critic
3 10
10
10
3 10
3 10
x 10
Actor Critic grad norm
N/A
N/A
N/A
0.5
Memory size
1 x 10
X 106
x 106
1 X 106
1 106
X 106
Entropy coefficient
N/A
N/A
0.2
N/A
N/A
0,01
Value loss coefficient
N/A
N/A
N/A
N/A
N/A
0.5
Exploration noisc
N/A
N/A
N/A
0.1)
N/A
N/A
Policy noise
N/A
N/A
N/A
(.2)
N/A
N/A
Noise clip
N/A
N/A
N/A
0.5
N/A
N/A
Use gae
N/A
N/A
N/A
N/A
True
True
Diffusion steps
20
100
N/A
N/A
N/A
N/A
Table 3:
used in QVPO.
Hyperparamelers
Ant-v3
HallCheetah-v3
Hopper-v3
Humanoid-v3
Waller2d-v3
Number of samples from diffusion policy Na 64
64
64
64
64
64
Number of samples from 2f a) N,
10
10
Action selection number for behavior policy Ko
4
4
Action selection number for target policy K,
2
1
Entropy weight want
0.01
0.01
0.01
0.01
0.01
Q-weight transformation function
gady
gadv
gady
qadv
qadv
C Training and Inference Time
The comparison of training and inference time is shown in the following tables. Notably, since the official implementation of QSM is based the jax and other algorithms based pytorch, is not fair comparison. In practice, the algorithm realized with jax is 6-10 times faster than that realized with pytorch. Besides, fairly compare the diffusion-based RL methods in training and inference time, we set the same number of diffusion steps (T=20) for all of them (i.e., QVPO, QSM, and DIPO). The results Table imply that QVPO can sufficiently use the parallel computing ability of GPU and the multiple-same sampling and selecting procedure does not take much time compared with gradient-based optimization like DIPO.
Table 4: The training and inference time comparison on Ant-v3 Benchmarks.
Method
QVPO
DIPO
TD3
SAC
PPO
SPO
QSM (jax)
Training Time (h)
6.8
10.5
0.5
0.3
1.0
Inference Time (ms)
6.2
0.2
0.3
0.9

---

Moreover, although QVPO is much slower than classical RL methods like SAC in inference, the inference time (6ms) still is acceptable. To our knowledge, most existing real robots only require 50-Hz control policy (i.e. output action per 20 ms). Besides, just like QSM, the inference time of QVPO can further improved with jax framework it necessary. Hence, the inference time not a bottleneck to applying QVPO to real applications.
18

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: As stated in the abstract and introduction, this paper proposes novel diffusion- based model-free online RL method, accurately reflecting the key contribution and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The analysis of limitations provided in Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
19

---

Justification:
The proofs are provided in Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: These details are provided in Appendix
B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification:
We will release the code once the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7/nips.com wttpcccpppbccccceec guwdes/ddeeeeee Cooeeeoodeesssseeeeeee /ttp://bb///uudddd///eeeeeeeesss
ssioiolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
These details are provided in Section and Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The standard deviation in experiment results (Figure and Table T shows the statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
21

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The information on the computer resources is provided in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https:
Answer: [Yes]
Justification:
Our research conformed with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our paper primarily focuses on theoretical research in how to employ the diffusion model in online reinforcement learning, with no consideration societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system Iearns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our experiment of this paper was conducted five MuJoCo locomotion tasks as Hopper-v3, Walker2d-v3. Ant-v3, HalfChectah-v3, and Humanoid-v? which poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The implementation of SAC, TD3, PPO, SPO, and DIPO cited properly in Appendix B Other assets are not applied here.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
23

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not introduce assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moddl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.


Reviews:
[{"review_id": "SvbxRI9rem", "reviewer": "Reviewer_nUs6", "summary": "The method proposes using Q-weight diffusion loss to train agents in online RL. Instead of approximating log probability, the paper uses advantage weight tuning to maintain the entropy of the actor.", "questions": "1. Equivalent Q-weight transformation functions are used to solve the negative Q value problem. Have the authors considered tuning rewards (for example, adding a fixed constant to all rewards to make all rewards positive) to guarantee Q values are all positive? It may be the most straightforward way to avoid the negative Q value issue.\n\n2. There is a typo in line 208 where the right parenthesis should be \"]\".\n\n3. For Eq(9), how do you estimate $\\max_a Q(s,a)$ in practice?\n\n4. Even though critic training is not the main point of the paper, I suggest adding a section in the Appendix to explicitly describe it for completeness.\n\n5. In Figure 5, some $K_b,K_t$ combinations may lead to extreme training collapse. Did the authors experiment with higher values of $K_b,K_t$ while maintaining $K_t<K_b$?", "limitations": "My main concern is that the design of equivalent Q-weight transformation functions is heuristic and may need some theoretical intuition or backup.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-organized, and the method is straightforward but effective. I found Theorem 1 interesting as it naturally builds a connection between diffusion models and the online RL objective. Experiments show the method is easy to converge and achieves higher rewards.", "weaknesses": "My main concern is that the paper has too much heuristic design regarding equivalent Q-weight transformation functions, the entropy regularization term, and Action Selection. Especially, Eq(9) and Eq(10) are called equivalent Q-weight transformation functions, but they are not mathematically equivalent to Eq(5).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1**: My main concern is that the design of equivalent Q-weight transformation functions is heuristic and may need some theoretical intuition or backup.\n\n**A1**: Thank you for raising the concern. Here is the theoretical proof to show the convergence of QVPO with qadv weight transformation function. (We are afraid OpenReview cannot display the equation clearly. You can copy the following equations as well as texts into HackMD for a clear presentation.) \n\nAssume the new diffusion policy after one update can be approximately expressed as \n$$\\pi_{new}(a\\mid s) \\approx (1-p_{data}(s)A_{\\pi_{old}}(s, a^\\star)\\eta)\\pi_{old}(a\\mid s) + p_{data}(s) A_{\\pi_{old}}(s, a^\\star) \\eta \\frac{\\mathbb{I}_{a\\in \\mathcal{N}(a^\\star\\mid s, \\epsilon)}(a)}{S_{\\mathcal{N}(a^\\star\\mid s, \\epsilon)}}\n$$\nwhere $a^\\star\\mid s$ is the action that can maximize $Q_{\\pi_{old}}(s, a)$ in the state $s$, $\\mathcal{N}(a^\\star \\mid s, \\epsilon)$ is the neighborhood of $a^\\star\\mid s$ with a small radius $\\epsilon$, $S_{\\mathcal{N}(a^\\star\\mid s, \\epsilon)}$ is the area of this neighborhood, and $p_{data}(s)$ is the sampling distribution of the state. This assumption is straightforward: the training sample's generation probability in the diffusion model will be improved and the improved probability is proportional to its weight.\n\nNow consider the improvement of the RL objective:\n$$\n\\begin{aligned}\n    \\mathcal{J}(\\pi_{new}) - \\mathcal{J}(\\pi_{old}) & = \\mathbb{E}_{s\\sim \\rho_0}\\left[V_{\\pi_{new}}(s) - V_{\\pi_{old}}(s)\\right] \\\\\n    & = \\mathbb{E}_{s\\sim \\rho_0}\\left[\\mathbb{E}_{a\\sim \\pi_{new}(a\\mid s)}\\left[Q_{\\pi_{new}}(s, a)\\right] - V_{\\pi_{old}}(s)\\right] \\\\\n    &=\\mathbb{E}_{s\\sim \\rho_0}\\left[\\mathbb{E}_{a\\sim \\pi_{new}(a\\mid s)}\\left[Q_{\\pi_{new}}(s, a)-Q_{\\pi_{old}}(s, a)\\right]\\right] \\\\ &\\quad \\quad  + \\mathbb{E}_{s\\sim \\rho_0}\\left[\\mathbb{E}_{a\\sim \\pi_{new}(a\\mid s)}\\left[ Q_{\\pi_{old}}(s, a)\\right] -  V_{\\pi_{old}}(s)\\right] \\\\\n    &=\\mathbb{E}_{s\\sim \\rho_0}\\left[\\mathbb{E}_{a\\sim \\pi_{new}(a\\mid s)}\\left[Q_{\\pi_{new}}(s, a)-Q_{\\pi_{old}}(s, a)\\right]\\right] + \\mathbb{E}_{s,a\\sim \\rho_0, \\pi_{new}(a\\mid s)}\\left[A_{\\pi_{old}}(s,a)\\right] \\\\\n\\end{aligned}\n$$\n\nThe first term here can be further expanded according to the Bellman equation:\n$$\n    \\mathbb{E}_{s\\sim \\rho_0}\\left[\\mathbb{E}_{a\\sim \\pi_{new}(a\\mid s)}\\left[Q_{\\pi_{new}}(s, a)-Q_{\\pi_{old}}(s, a)\\right]\\right] = \\gamma \\mathbb{E}_{s\\sim d^1_{\\pi_{new}}}\\left[V_{\\pi_{new}}(s)-V_{\\pi_{old}}(s)\\right]\n$$\nwhere $d^1_{\\pi_{new}}$ denotes the probability distribution of state in time step $t=1$ with policy $\\pi_{new}$. Repeating the above operation, we will obtain:\n\n$$\n\\begin{aligned}\n\\mathcal{J}(\\pi_{new}) - \\mathcal{J}(\\pi_{old}) &= \\sum_{t=0}^\\infty \\gamma^t\\mathbb{E}_{s,a \\sim d_{\\pi_{new}}^t, \\pi_{new}}\\left[A_{\\pi_{old}}(s,a)\\right] \\\\\n&= \\frac{1}{1-\\gamma} \\mathbb{E}_{s \\sim d_{\\pi_{new}}}\\left[\\mathbb{E}_{a\\sim \\pi_{new}(\\cdot\\mid s)}\\left[A_{\\pi_{old}}(s,a)\\right]\\right] \\\\\n&\\approx \\frac{1}{1-\\gamma} \\mathbb{E}_{s \\sim d_{\\pi_{new}}}\\Bigg[(1-p_{data}(s)A_{\\pi_{old}}(s,a^\\star)\\eta)\\mathbb{E}_{a\\sim \\pi_{old}(\\cdot\\mid s)}\\left[A_{\\pi_{old}}(s,a)\\right] \\\\& \\quad \\quad + p_{data}(s)A^2_{\\pi_{old}}(s,a^\\star) \\eta \\frac{\\mathbb{I}_{a\\in \\mathcal{N}(a^\\star\\mid s, \\epsilon)}(a)}{S_{\\mathcal{N}(a^\\star\\mid s, \\epsilon)}}\\Bigg] \\\\\n& = \\frac{1}{1-\\gamma} \\mathbb{E}_{s \\sim d_{\\pi_{new}}}\\left[p_{data}(s)A^2_{\\pi_{old}}(s, a^\\star) \\eta \\frac{\\mathbb{I}_{a\\in \\mathcal{N}(a^\\star\\mid s, \\epsilon)}(a)}{S_{\\mathcal{N}(a^\\star\\mid s, \\epsilon)}}\\right] \\ge 0\n\\end{aligned}\n$$\n\n\n> **Q2**: Have the authors considered tuning rewards (for example, adding a fixed constant to all rewards to make all rewards positive) to guarantee Q values are all positive? \n\n**A2**: Thank you for your question. Yes, we did conduct such a test by adding a fixed constant to all rewards at the very beginning. However, it does not work well after the policy has been improved to a certain extent. This is because the relative difference of Q value will be reduced with this operation. For example, if we have $Q(s,a_1)=1, Q(s,a_2)=2$, and $Q^{'}(s,a_1)=11, Q^{'}(s,a_2)=12$ after adding a fixed constant $10$, the raw relative difference is $\\frac{2-1}{2}=0.5$ and the modified relative difference is $\\frac{12-11}{12}=0.083$. Besides, in many real applications, how to set this fixed constant to ensure all $Q$ values positive is hard. \n\n\n> **Q3**: There is a typo in line 208 where the right parenthesis should be \"]\".\n\n**A3**: Thank you for pointing out this. We will correct this typo in the final version.\n\n> **Q4**: For Eq(9), how do you estimate $\\max_a Q(s,a)$ in practice?\n\n**A4**: As mentioned in lines 219-220, we can obtain the approximate $\\max_a Q(s,a)$ via enough samples from the diffusion. In practice, we choose around 64 samples. \n\n> **Q5**: Even though critic training is not the main point of the paper, I suggest adding a section in the Appendix to explicitly describe it for completeness.\n\n**A5**: Thank you for your valuable suggestion. The critic training part is standard and the same as TD3 and DiPO. We will add the critic training part to the appendix in the final version. \n\n> **Q6**: In Figure 5, some $K_b, K_t$ combinations may lead to extreme training collapse. Did the authors experiment with higher values of $K_b, K_t$ while maintaining $K_t<K_b$? \n\n**A6**: Thank you for your constructive suggestion. Firstly, we need to clarify that the fluctuation of the reward curve is a common phenomenon in RL and cannot be viewed as a training collapse, since it recovers quickly to the previous level. In Figure 5, we only plot the result of one run for each case without a window smoothing. That is why the reward curves look unstable. \n\nWe have conducted experiments with higher $K_b$ and have presented the results in the attached PDF file. It can be found that a high value of $K_b$ will limit the exploration of QVPO and tend to be stuck at a local optimum.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EI6hBMZmcQ", "reviewer": "Reviewer_Kypj", "summary": "The paper uses a diffusion policy to address online RL. The method works by weighting the diffusion model VLO loss using advantages computed with a Q function. Additionally, they add an entropy maximization term to aid with exploration. The authors present results indicating strong performance on a variety of continuous control tasks as well as ablations verifying the necessity of several components of their method.", "questions": "Random questions and comments\n - The second and third sentence of the abstract seem to contradict one another. How is it that diffusion policies can improve exploration (which is only relevant to online RL) if they have only really been applied to offline RL?\n - In the introduction you claim that DIPO has issues with limited exploration but I don't see any proof for that claim in your or their paper\n - In the introduction you claim that QSM has issues that prevent it from converging to optimal policies that your method does not have, but you never compare to QSM to demonstrate this win. I think QSM is an important comparison that really should be added to the paper since it is probably the closest to your method.\n - I don't understand the purpose of including qcut in the manuscript since you immediately propose a different approximation that works better and then never mention qcut again.\n - I'm very confused by Figure 2. How did you generate this? What does 'explorable area' mean? \n - Section 4.4 seems to imply that selecting an action by sampling several and then choosing the one that maximizes the Q value is a novel contribution, but this has been done on several occasions including by Diffusion QL, Efficient Diffusion Policy, Implicit Diffusion Q Learning, etc. This should be rewritten and these works should be cited. Also it is frequently referred to as \"efficient diffusion policy\" in the paper which is confusing since there is another paper (Kang et al.) that is called the same thing.\n - The policy improvement objective in this paper is actually very similar to one of the variants of EDP (Kang et al.), specifically shown in Equation 13 of https://arxiv.org/pdf/2305.20081. This paper still has reasonable novelty compared to that paper but that similarity should be better acknowledged.\n - Figure 4 is a nice result but it would be helpful to also see steps=100 & ent.\n - I'm curious about the wall clock training time of QVPO compared with baselines. Does everything take about 10 hours? I would expect QVPO to be quite a bit slower.\n - Is $\\omega_\\textrm{ent}$ difficult to tune?\n\n\n\nNit picks (not penalizing you for these but thought I should point them out):\n - The text in the plot legend is too small and the plots would be easier to read if the authors applied some smoothing\n - The transitions used at the beginning of the sentences in lines 163-170 (besides, firstly, notably, etc) are very distracting and the paragraph would sound better if you removed all of them.\n - There is an extra \" on line 237 for some reason\n - Appendix B: hyperparameters should not be hyphenated\n - The legends of the plots look sloppy. Matplotlib allows the use of LaTeX and you should make use of that", "limitations": "The authors mention one limitation--the lack of adaptive entropy regularization. However, they miss other potential limitations, such as slower wall clock training time or the inability to output extended chunks of actions, as is common for diffusion policies applied to BC.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is one of the first to consider the application of diffusion policies to online RL and gets reasonably good results. This could be a significant paper with some additional experiments and touch ups.\n - The paper is the first to apply the weighted VLO objective to online RL and contributes a novel method for maximizing the entropy of a diffusion policy\n - Overall the discussion of the method and experiments is clear", "weaknesses": "There are several claims made in the paper that I don't believe are substantiated. Some important ones:\n - In the introduction and RW section you make claims about the DIPO and QSM baselines that don't seem to be proven anywhere\n - You take credit for several things that aren't yours. Your \"efficient diffusion policy\" trick is used extensively in other work and the weighted VLO loss is one of the methods presented in EDP (Kang et al.)\n\nI think there need to be some changes to the experiments.\n - Most important: the comparison to other RL algorithms is unfair since you tune your algorithm per environment whereas other algorithms have a fixed set of hyperparameters. You need to either tune all relevant hyperparameters for all other algorithms for all environments or fix yours.\n - One of your key claims is that the diffusion policy is a good fit for RL because of its ability to model multimodal action distributions and yet you don't consider any environments where this is the case. Rather, you simply demonstrate improved sample efficiency on a handful of tasks that clearly can be solved using a MLP policy parameterization\n - You really should compare to QSM\n - The ablation studies are a bit incomplete in my opinion. It would be good to see results from all five environments. Also, it would be good to run an ablation where you omit the policy loss weighting, leading to an algorithm similar to IDQL, to verify that the policy weighting is important and that your policy isn't relying on the efficient diffusion policy step. I assume this ablation will go well for you based on the fact that the $K_b=1$ ablation still learns a policy but this will strengthen the paper.\n - You mention qcut, then say it doesn't work and never mention it again. Either show an experiment proving that or remove it from the paper\n - I believe that slow wall clock time, either for training or online rollouts, is potentially a big limitation of the method. Wall clock time is vaguely mentioned in the appendix but it would be good to see some data about this in the paper", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1**: You make claims about DIPO and QSM that don't seem to be proven anywhere. You really should compare to QSM. \n\n**A1(DIPO)**: **We did compare QVPO with DIPO in Figure 4 (original paper)**. Besides, as mentioned in lines 110-114, DIPO creates a dedicated buffer for diffusion policy, updates the state-action pairs in this buffer via the gradient of Q network, and finally uses the state-action pair from this buffer to train the diffusion policy. In that case, the training samples of DIPO will be limited to the vicinity of previously explored actions, thus leading to limited exploration. \n\n**A1(QSM)**: The **code of QSM was not yet released when this paper was submitted**. So we did not compare QVPO with QSM before. We have **added the QSM results** of 5 Mujoco Benchmarks in Figure 2 of the attached pdf file, following the **recently released** official implementation. We can see the performance of QSM is much worse than QVPO and DIPO, and even worse than SAC and TD3 in the Walker and Humanoid environments, **which is also shown in Figure 3 of the QSM original paper**. As we mentioned in lines 43-46, QSM has a doubled approximation error from the alignment process. The trained Q value function has an approximation error. QSM utilizes the gradient of this approximated Q function to train the score model. \n\n> **Q2**: You take credit ... \"efficient diffusion policy\" trick is used extensively in other work and the weighted VLO loss is one of the methods presented in EDP (Kang et al.) ... This should be rewritten and these works should be cited. ... This paper still has reasonable novelty but that similarity should be better acknowledged.\n> \n**A2**: We will add more illustrations and cite these works properly in the final version. Note that these works are all offline RL. They used \"efficient diffusion policy\" to yield deterministic policy during inference, while our QVPO utilizes \"efficient diffusion policy\" in the online interaction with the environment to obtain **more efficient transition data**. Our motivation is different from these existing works. \n\nBesides, the weighted VLO loss of QVPO is different from EDP. In the weighted VLO loss, QVPO trains diffusion policy with the weighted sample from the current policy, while EDP trains the diffusion policy with the weighted sample from policy in the offline dataset (replay buffer). \n\n> **Q3-1**: The comparison to other RL algorithms is unfair. Need to either tune all relevant hyperparameters for other algorithms or fix yours. \n\n**A3-1**: Most hyperparameters in our experiments are the same, except $K_t$ in Hopper and $w_{ent}$ in HalfCheetah, as shown in Table 3. During the rebuttal, we **have fixed all hyperparameters for QVPO** and the **new results** are shown in the **attached pdf file for 5 Mujoco benchmarks**. QVPO shows similar performance and still outperforms all baselines with a clear margin. \n\n>**Q3-2**: **Is $\\omega_{ent}$ difficult to tune?** \n\n**A3-2**: No, quite easy to set this value. Just make the scaling within 1e-3 to 1e-2. \n\n> **Q4**: Claim diffusion policy is a good fit for RL due to its ability to model multimodal action distributions, but don't conduct experiments on such environment. \n\n**A4**: Firstly, existing online RL methods almost all verify their performance on these robot locomotion tasks. In that case, we can compare QVPO with previous methods in a relatively fair manner with these standard environments. \n\nSecond, the multimodality of policy is important because it can avoid the policy stuck at the local optimum. Moreover, our tested tasks such as Humanoid-v3 cannot be solved very well with a MLP policy. \n\nThird, to further test QVPO for more complex tasks, we applied QVPO to the recently released HumanoidBench environment. The benchmark is based on Unitree H1 humanoid robot with **151 state dimension** and **61 action dimension**. Please refer to **Global Response** for more details. \n\n> **Q5(1)**: The ablation studies are a bit incomplete in my opinion. It would be good to see results from all five environments. \n \n**A5(1)**: Due to the time and resource limit, we cannot complete these additional experiments. We will add them in our final version. Actually, the results for other 4 environments are similar but we did not record them. \n \n> **Q5(2)**: It would be good to run an ablation where you omit the policy loss weighting ... I assume this ablation will go well ... $K_b=1$ still learns a policy but this will strengthen the paper. \n\n**A5(2)**: The requested ablation study $K_b=1$ is shown in Figure 5 of the original paper. Offline diffusion RL methods like EDP train the diffusion policy with samples from offline datasets or replay buffer, while QVPO trains the diffusion policy with selected samples from the diffusion policy itself.  \n\n> **Q6**: You mention qcut. Either show an experiment proving that or remove it. \n\n**A6**: In the beginning, qcut is mentioned to present the original motivation of QVPO. We will remove qcut part in the final version. \n\n> **Q7**: The training and inference time of QVPO. \n\n**A7**: Please refer to the **Global Response**. \n\n> **Q8**: The second and third sentence of the abstract seem to contradict one another. \n\n**A8**: Thanks and we will modify the two sentences. \n\n> **Q9**: I'm very confused by Figure 2. How did you generate this? What does 'explorable area' mean?\n\n**A9**: Figure 2 is exported by powerpoint to visualize the effect of diffusion entropy regularization term. The 'explorable area' means the area with a certain generation probability of diffusion policy.  \n\n> **Q10**: Figure 4 is a nice result but it would be helpful to also see steps=100 & ent.\n\n**Q10**: The entropy term is introduced for the diffusion model with few denoising steps. QVPO with steps=100 with entropy term will not improve the result. We will add this result in the final version. \n\n> **Q11**: Suggestions to improve the presentation (some minor format issues) ...\n\n**A11**: We will modify them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y0GZul2UrV", "reviewer": "Reviewer_Qp9e", "summary": "The paper proposes a novel model-free online reinforcement learning (RL) algorithm called Q-weighted Variational Policy Optimization (QVPO), which leverages the expressiveness and multimodality of diffusion models. By introducing Q-weighted variational loss and entropy regularization, the authors aim to overcome the limitations of unimodal policies and enhance exploration capabilities. The algorithm is validated through experiments on MuJoCo continuous control benchmarks, demonstrating state-of-the-art performance in terms of both cumulative reward and sample efficiency.", "questions": "1. Can the proposed algorithm be adapted for tasks with discrete action spaces or sparse rewards, and if so, how?\n2. How is the diffusion policy initialized, and what impact does this initialization have on the performance of the algorithm? If the policy is tranined from scratch, how is the Q-function trained at the beginning stage?\n3. The efficient diffusion policy method reduces policy variance but may limit exploration. How do you balance exploration and exploitation, and can you provide an ablation study to verify the effects of this trade-off?\n4. From Figure 3, QVPO appears to converge faster. Can you provide an explanation or experiments to support this observation?\n5. How does the computational complexity of QVPO compare with existing diffusion-based online RL methods in terms of training time and resource utilization?", "limitations": "The authors have addressed the limitations of the proposed method.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper presents a significant advancement in integrating diffusion models into online RL, a relatively underexplored area. The proposed QVPO algorithm is theoretically sound, with the Q-weighted variational loss being proven as a tight lower bound of the policy objective in online RL. The incorporation of an entropy regularization term and an efficient behavior policy are innovative approaches to enhance exploration and sample efficiency.", "weaknesses": "1. Experiment Scope: The experiments are limited to D4RL datasets, which do not cover the full spectrum of possible RL environments. The generalization of the proposed method to other types of tasks, such as discrete action spaces or tasks with sparse rewards, is not demonstrated, eg. Adroit, Humanoid, MetaWorld, Maniskill, KUKA Pick-and-Place.\n2. Ablation Studies: While the paper includes ablation studies, they are not exhaustive. Additional ablation studies are needed to isolate the contributions of individual components, such as the Q-weighted variational loss and the entropy regularization term. It is unclear how much each component independently contributes to the overall performance improvements.\n3. Diffusion Policy Variance Reduction: The proposed method for reducing diffusion policy variance via action selection aims to improve sample efficiency. However, this approach may sacrifice the exploration capabilities of the diffusion policy, thus limiting its multimodal potential. This trade-off is not discussed in the paper, and no ablation study is provided to evaluate the impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for his/her careful reading and valuable suggestions. Below we will answer your concerns point-by-point.\n\n> **Q1**: The experiments are limited to D4RL datasets, which do not cover the full spectrum of possible RL environments. Can the proposed algorithm be adapted for tasks with discrete action spaces or sparse rewards, and if so, how?\n\n**A1**: Thanks for your question. We need to clarify two fundamental points: \n\n(1) **We never use D4RL datasets. QVPO is an online RL method instead of an offline one. Only offline RL requires datasets such as D4RL.** We perform our experiments in 5 mujoco online environments rather than D4RL datasets. These mujoco tasks are quite standard benchmarks in existing online RL methods. Besides, we also added the experiments on the recently released HumanoidBench environment in the attached pdf file. It is a very complex environment and QVPO still achieves the SOTA performance. \n\n(2) **QVPO is designed to handle complex continuous control tasks rather than for discrete action spaces or sparse rewards.** For discrete action spaces, we can directly sample in the discrete probability simplex output by the neural network and achieve multimodal policy. However, it is not trivial to achieve multimodal policy in continuous action distribution, since we cannot output the corresponding probability density function via neural networks. In that case, we consider utilizing diffusion model to enhance the multimodality and exploration ability of RL agent in continuous control tasks. Besides, QVPO does not conflict with most existing RL methods for sparse rewards like HER [R5]. Thus, we can incorporate HER into QVPO in environments with sparse rewards.\n\n> **Q2**:  While the paper includes ablation studies, they are not exhaustive. Additional ablation studies are needed to isolate the contributions of individual components, such as the Q-weighted variational loss and the entropy regularization term. It is unclear how much each component independently contributes to the overall performance improvements.\n\n**A2**: Thanks for raising the concern. We need to clarify that the main body of QVPO is the Q-weighted variational loss and QVPO must use this loss to train the diffusion policy. Thus, there is no way to do ablation study on the Q-weighted variational loss. Besides, the ablation study on the entropy regularization term is shown in Figure 4 of the original paper.\n\n\n> **Q3**: The proposed method for reducing diffusion policy variance via action selection aims to improve sample efficiency. However, this approach may sacrifice the exploration capabilities of the diffusion policy, thus limiting its multimodal potential. This trade-off is not discussed in the paper, and no ablation study is provided to evaluate the impact.\n\n**A3**: We did conduct such ablation study with the trade-off analysis in our paper. The action selection number $K_b$ in efficient diffusion policy is actually the trade-off between exploration and exploitation. As shown in **Figure 5** and mentioned in lines 320-330, the diffusion policy converges slowly when $K_b=1$ (i.e., without efficient diffusion policy). This means QVPO without efficient diffusion policy cannot exploit the information of Q value very well. Besides, we also add the experiment with $K_b=20$ to show that too high action selection number will lead to a limited exploration. In that case, setting $K_b=4$ can balance exploration and exploitation well. \n\n> **Q4**: How is the diffusion policy initialized, and what impact does this initialization have on the performance of the algorithm? If the policy is trained from scratch, how is the Q-function trained at the beginning stage?\n\n**A4**: Thanks for your question. The diffusion policy and Q-function are both trained from scratch without any special initilization trick.\n\n> **Q5**: The efficient diffusion policy method reduces policy variance but may limit exploration. How do you balance exploration and exploitation, and can you provide an ablation study to verify the effects of this trade-off?\n\n**A5**: The answer can be referred to in **A3**.\n\n\n> **Q6**: From Figure 3, QVPO appears to converge faster. Can you provide an explanation or experiments to support this observation?\n\n**A6**: Thanks for your question. Compared with DIPO and QSM, QVPO can explore broader action space and achieve multimodal training samples with the sampling-based update and entropy regularization term. We think that is why QVPO converges faster.\n\n> **Q7**: How does the computational complexity of QVPO compare with existing diffusion-based online RL methods in terms of training time and resource utilization?\n\n**A7**: The training and inference time of QVPO and other online RL methods are shown in **Global Rebuttal**. The GPU resource utilization rate of QVPO/DIPO is 55%/28% and memory consumption of QVPO/DIPO is 598MB/1354MB. In that case, QVPO can sufficiently make use of the parallel computing ability of GPU. \n\n[R5] Andrychowicz M, Wolski F, Ray A, et al. Hindsight experience replay[J]. Advances in neural information processing systems, 2017, 30.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eEW7s8Leub", "reviewer": "Reviewer_uSjP", "summary": "This paper introduces QVPO, a new model-free algorithm that trains a diffusion policy online. It proposes a Q-weighted VLO loss by weighting the original diffusion model objective with Q-values. To address the issue of negative Q-values, the algorithm uses advantage instead. QVPO also encourages exploration by mimicking the uniform distribution and reduces inference variance by executing actions with higher Q-values.", "questions": "- Can the author provide an explanation for why the exploration ability of diffusion policies declines with a limited number of diffusion steps? Additionally, what happens when the diffusion steps are even fewer, say 5, a common value in other diffusion policy literature? I suppose the entropy term should perform better in such cases.\n\n- If the maximum entropy term is used throughout the training, the policy may continue to follow the uniform distribution in the later stages, potentially hurting performance and convergence. If this is the case, would a scheduling mechanism improve the final performance?\n\n- Can the author explain why the Q-weighted VLO loss is superior to QSM regarding the additional errors in policy optimization?\n\n- In line 300, it is mentioned that action selection is used during inference, which is a standard technique in offline RL. However, is the same technique applied to the baselines? If not, this would be unfair.", "limitations": "--", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- The paper addresses a relevant problem, which is to learn diffusion policy online.\n\n- The paper is generally well-written and easy to read. \n\n- The proposed method demonstrates impressive results on MuJoCo benchmarks compared to SOTA baselines.", "weaknesses": "- Several claims require additional support. For instance, while Figure 2 is informative in demonstrating the ideal effect of the entropy term, experimental results on toy examples would provide stronger evidence than an illustration.\n\n- Since QVPO relies on action samples for both optimization and action selection, this will increase computational demands.\n\n- The tasks are limited and relatively trivial, making it difficult to tell performance in more complex tasks, such as robot manipulation. In particular, when the action space is larger, it is unclear how the entropy term will scale given its reliance on uniform action distributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your valuable feedback and comments! We itemize the weaknesses you mentioned and answer them.\n\n> **Q1**: While Figure 2 is informative in demonstrating the ideal effect of the entropy term, experimental results on toy examples would provide stronger evidence than an illustration.\n\n**A1**: We perform a toy experiment on a continuous bandit problem to further present the effect of the entropy term. The results are shown in Figure 1 of the attached pdf file, which is similar to the illustration. We will add it in the final version. \n\n> **Q2**: Since QVPO relies on action samples for both optimization and action selection, this will increase computational demands.\n\n**A2**: Actually, that is not the weakness but the advantage of QVPO. QVPO can sufficiently utilize the parallel computing ability of GPU via the multiple sampling and selection procedure. We observed that the previous diffusion-based RL methods generally have a low gpu utilization rate since the computational bottleneck of the forward process in the diffusion. In that case, parallelly sampling multiple samples does not add much extra time cost. \n\nBesides, QVPO does not need to use the gradient from Q network to optimize the diffusion policy. This also reduces the time cost to a certain extent compared with other diffusion RL methods like DIPO. Finally, the comprehensive training time and inference time comparison can be referred to in **Global Rebuttal**.\n\n> **Q3**: The tasks are limited and relatively trivial, making it difficult to tell performance in more complex tasks, such as robot manipulation. In particular, when the action space is larger, it is unclear how the entropy term will scale given its reliance on uniform action distributions.\n\n**A3**: We need to clarify that these 5 mujoco environments are the standard online RL benchmarks and most of the previous online RL methods (e.g., SAC, DIPO, etc.) only choose them to do the experiments. That is why we verify our algorithm on them. \n\nBesides, to further present the efficiency of QVPO, we also apply it to the recently released HumanoidBench environment, which is a complex continuous control benchmark based on Unitree H1 humanoid with **151 observation dimension** and **61 action dimension**. The results are shown in Figure 3 of the attached pdf file. It can be observed that model-free RL methods (SAC, PPO) do not work totally, while QVPO (as a model-free method) achieves superior performance compared with advanced model-based methods (Dreamer-v3, TD-MPC2). \n\nThe entropy coefficient is set 0.01 here as well. In fact, we found that our algorithm is robust enough with the scaling of $w_{ent}$ in 1e-2 to 1e-3. \n\n> **Q4(1)**: Can the author provide an explanation for why the exploration ability of diffusion policies declines with a limited number of diffusion steps? \n\n**A4(1)**: Note that the Gaussian noise is added at each diffusion denoising step. In other words, more diffusion steps lead to more possibilities. Thus, the exploration ability of diffusion policies declines with a limited number of diffusion steps. \n\n> **Q4(2)**: Additionally, what happens when the diffusion steps are even fewer, say 5, a common value in other diffusion policy literature? I suppose the entropy term should perform better in such cases.\n\n**A4(2)**: We add the result of the ablation study with 5 diffusion steps in the attached pdf file. It can be found that the performance is poor. Hence, different from offline RL methods, a certain number of diffusion steps is required in online RL setting. This is also verified in previous works like DIPO [R7].\n\n> **Q5**: If the maximum entropy term is used throughout the training, the policy may continue ... potentially hurting performance and convergence. Would a scheduling mechanism improve the final performance?\n\n**A5**: The entropy term is not constant as you thought. As mentioned in Algorithm 1, line 8. You can find that the actual entropy coefficient $w_{ent}(s)=w_{ent}w_{eq}(s,a_{max})$ is related to $w_{eq}(s,a_{max})=A(s,a_{max}), A(s,a_{max})\\ge 0$. When QVPO converges, $A(s,a_{max})\\rightarrow 0$ and $w_{ent}(s)\\rightarrow 0$ . \n\n> **Q6**: Can the author explain why the Q-weighted VLO loss is superior to QSM regarding the additional errors in policy optimization?\n\n**A6**: As mentioned in lines 43-46, QSM has a doubled approximation error from the alignment process. More concretely, the trained Q value function has an approximation error. QSM utilizes the gradient of this approximated Q function to train the score model. This leads to the doubled approximation error in the score model. That is why the performance of QSM is worse than QVPO as shown in the attached pdf. Besides, the policy improvement stage of DIPO is sampling-based, which implies QVPO is more likely to cross the local optimum.\n\n> **Q7**: In line 300, it is mentioned that action selection is used during inference, which is a standard technique in offline RL. However, is the same technique applied to the baselines? If not, this would be unfair.\n\n**A7**: We need to clarify that there is a contradiction if we apply this technique to other methods. \n\nAs mentioned in offline RL works like IDQL [R6], applying action selection to diffusion policy is to yield a deterministic policy during inference. For MLP or Gaussian policy, we can directly output the deterministic action or the mean of Gaussian policy to achieve this goal. Hence, this trick is not appropriate for classical online RL methods (e.g., SAC, TD3). \n\nFor diffusion-based RL, like DIPO, it yields the deterministic policy via fixing the denoising process (i.e., updating with the mean value in each diffusion step). In that case, there exists a contradiction if action selection is also applied during inference. \n\n[R6] Hansen P, et al. Idql: Implicit q-learning as an actor-critic method with diffusion policies.\n\n[R7] Yang L, et al. Policy representation via diffusion probability model for reinforcement learning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their careful reading of considerate and meaningful suggestions to help us improve our paper. We sincerely appreciate that the reviewers find our work "straightforward but effective" (nUs6), "a significant work" (Kypj, Qp9e), "novel and innovative as the first to apply the weighted VLO loss to online RL" (Kypj, Qp9e), and contributes "a novel method for maximizing the entropy of a diffusion policy" (Kypj). We are further glad that the reviewers agree unanimously that our manuscript is "well-written and easy to read" (nUs6, Kypj) and confirm our contributions on both the "interesting and sound" theoretical analysis and "reasonably good, impressive, easy to converge" empirical results (Kypj, Qp9e, uSjP, nUs6) to support our algorithm.

In the following, we will try to address the concerns/questions of the reviewers and present a detailed item-by-item response to their comments.

**(1) Illustration of the attached pdf file**: The attached pdf file contains 5 figures. 

Figure 1 replaces the schematic diagram in the original paper with an experiment on a continuous bandit toy example according to Reviewer uSjP. The contour lines indicate the reward function of continuous bandit, which is an arbitrarily selected function with 3 peaks. The concrete reward function is $Q(x)=\sum_{i=1}^3w_i\frac1{2\pi\sigma_i^2}\exp\left(-\frac1{2\sigma_i}(x-\mu_i)^T(x-\mu_i)\right)$, where $w_i=1.5$, $\sigma_i=0.1$, and $\mu_i=[-1.35,0.65]^T$, $[-0.65,1.35]^T$ and $[-1.61,1.61]^T$ respectively.

Figure 2 adds QSM as a new comparison baseline and shows rerun experimental results on QVPO with fixed hyperparameters. The legends and plots are all improved according to the suggestions of Reviewer Kypj.

Figure 3 adds experimental results on the recently released HumanoidBench [R4]. The benchmark is very complex based on Unitree H1 humanoid robot with **151 observation dimension** and **61 action dimension**, and **most existing model-free RL methods do not work totally in this environment.**  It verifies QVPO is competitive even compared with advanced model-based methods (e.g., TD-MPC2 [R1], Dreamer-v3 [R2]) in complex continuous control tasks that have high state and action dimensions. 

Figure 4 is the comparison between QVPO with and without entropy term, which adds the experimental result of 5 diffusion steps with entropy term according to Reviewer uSjP. We believe 5 diffusion steps are not enough for diffusion policy in online RL.   

Figure 5 is the comparison of QVPO with different action selection numbers for behavior policy $K_b$ and for target policy $K_t$, which adds the new case with $K_b=20, K_t=2$ according to Reviewer Qp9e. It implies that a too high $K_b$ in QVPO will result in a lack of exploration.

**(2) Comparison on training time and inference time**:
The training and inference time comparison is shown in the following tables. **Notably, since the official implementation of QSM is based on the jax and other algorithms are based on pytorch, it is not a fair comparison.** In practice, the same algorithm realized with jax is 6-10 times faster than that realized with pytorch. Besides, to fairly compare the diffusion-based RL methods in training and inference time, we set the same number of diffusion steps (T=20) for all of them (i.e., QVPO, QSM, and DIPO). The results imply that QVPO can sufficiently use the parallel computing ability of GPU and the multiple-sampling and selecting procedure does not take much time compared with gradient-based optimization like DIPO. 

Moreover, we need to clarify that: **although QVPO is much slower than classical RL methods like SAC in inference, the inference time (6ms) still is acceptable**. To our knowledge, most existing real robots only require a 50-Hz control policy (i.e. output action per 20 ms). Besides, just like QSM, the inference time of QVPO can be further improved with jax framework if it is necessary. Hence, the inference time is not a bottleneck to applying QVPO to real applications.

It is worth noting that we only run one program at the same time here to avoid the effect of other running programs. That is why the training time of QVPO is less than what we mentioned in the original paper.

Due to the word limit, here we only show the comparison on Ant-v3. Results on other 4 environments are almost the same.

The training time comparison on Ant-v3 Benchmarks.
|  Method            |    QVPO   |     DIPO  |    TD3     |      SAC    |     PPO       |      SPO      | QSM (jax)  |
|:------------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|
| **Training Time (h)**  |   6.8    |    10.5  |     0.5   |        2.5     |     0.3     |   0.3  |    1.0        |

The inference time comparison on Ant-v3 Benchmarks.
|  Method            |    QVPO   |     DIPO   |    TD3     |      SAC    |     PPO       |      SPO      | QSM (jax)  |
|:------------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|
| **Inference Time (ms)**  |   6.2    |    5.7    |      0.2  |        0.3     |     0.2      |    0.3  |    0.9        |

[R1] Hansen N, Su H, Wang X. Td-mpc2: Scalable, robust world models for continuous control[J]. arXiv preprint arXiv:2310.16828, 2023. 

[R2] Hafner D, Pasukonis J, Ba J, et al. Mastering diverse domains through world models[J]. arXiv preprint arXiv:2301.04104, 2023.

[R3] Peng X B, Kumar A, Zhang G, et al. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning[J]. arXiv preprint arXiv:1910.00177, 2019.

[R4] Sferrazza C, Huang D M, Lin X, et al. Humanoidbench: Simulated humanoid benchmark for whole-body locomotion and manipulation[J]. arXiv preprint arXiv:2403.10506, 2024.

Meta-review: The reviewers appreciate the novelty of the work as one of the first results applying diffusion policies to online RL and obtaining reasonable results. However, they were all concerned about some limitations in the experiments and several claims made in the paper. Some of these concerns were addressed during the rebuttals and as a result reviewers raised their scores. However, it is still a borderline paper, slightly above the bar. I believe the novelty of the approach could be a good reason for acceptance but I strongly recommend that the authors take the questions/concerns raised by the reviewers into account in preparing the next revision of their work. Especially, it would be great if they can run some experiments in more difficult environments.
Decision: Accept (poster)