Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization

Authors: shutong  ding
ke  hu
zhenhao  zhang
kan  ren
weinan  zhang
jingyi  yu
jingya  wang
ye  shi
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 14:35:30

Abstract: Diffusion models have garnered widespread attention in Reinforcement Learning (RL) for their powerful expressiveness and multimodality. It has been verified that utilizing diffusion policies can significantly improve the performance of RL algorithms in continuous control tasks by overcoming the limitations of unimodal policies, such as Gaussian policies. Furthermore, the multimodality of diffusion policies also shows the potential of providing the agent with enhanced exploration capabilities. However, existing works mainly focus on applying diffusion policies in offline RL, while their incorporation into online RL has been less investigated. The diffusion model's training objective, known as the variational lower bound, cannot be applied directly in online RL due to the unavailability of 'good' samples (actions). To harmonize the diffusion model with online RL, we propose a novel model-free diffusion-based online RL algorithm named Q-weighted Variational Policy Optimization (QVPO). Specifically, we introduce the Q-weighted variational loss and its approximate implementation in practice. Notably, this loss is shown to be a tight lower bound of the policy objective. To further enhance the exploration capability of the diffusion policy, we design a special entropy regularization term. Unlike Gaussian policies, the log-likelihood in diffusion policies is inaccessible; thus this entropy term is nontrivial. Moreover, to reduce the large variance of diffusion policies, we also develop an efficient behavior policy through action selection. This can further improve its sample efficiency during online interaction. Consequently, the QVPO algorithm leverages the exploration capabilities and multimodality of diffusion policies, preventing the RL agent from converging to a sub-optimal policy. To verify the effectiveness of QVPO, we conduct comprehensive experiments on MuJoCo continuous control benchmarks. The final results demonstrate that QVPO achieves state-of-the-art performance in terms of both cumulative reward and sample efficiency.

Main paper: 
 introduction: 
 Recent years have witnessed significant success in the application of diffusion policy in imitation learning [6, 22, 27, 32] , and offline reinforcement learning [40, 1, 16, 4, 5, 11, 44, 26] . In imitation learning, diffusion models [13, 35, 7] are often employed to capture the intricate distributions found within expert datasets. Furthermore, in offline RL, certain works replace unimodal policies with diffusion models to enrich the diversity in sampled actions or decision sequences. This success can be largely attributed to the robust expressiveness and multimodality inherent in diffusion models [13, 35] . Such methods have demonstrated their effectiveness in practical scenarios like robotic navigation [36, 43] , robot arm manipulation [6, 17, 33, 23] , dexterous hand and legged robot locomotion control [14] .While diffusion policies have been extensively explored, their application in online RL has received relatively less attention. In online RL, value estimation varies with policy changes [45] , posing a significant challenge. While diffusion models can effectively capture data distribution, they cannot directly improve the policy due to their training objectives [42] . Consequently, integrating diffusion models into traditional online RL framework [34, 10, 9, 38] is challenging. Some works, such as DIPO [42] , propose leveraging Q-function gradients to update actions for higher rewards, followed by employing diffusion models to fit the updated action distribution. However, relying on gradient updates constrains the algorithm's exploration capability. QSM [31] directly aligns the score and the gradient of the Q-function under the perspective of score-matching. However, since the gradient of the Q-network is inaccurate, QSM has a doubled approximation error from the alignment process, which prevents the policy from converging to optimality.To address these issues, we propose a novel model-free online RL algorithm called Q-weighted Variational Policy Optimization (QVPO). The core idea behind QVPO is straightforward yet effective. By revisiting the VLO objective of diffusion models and the policy objective of online RL, we discovered that the Variation LOwer Bound (VLO) objective, with appropriate weights for state-action pairs, can form a tight lower bound of the policy objective under certain conditions. This new objective, termed Q-weighted VLO loss, is complemented by Q-weight transformation functions, allowing it to be applied to general RL tasks. Additionally, we add an entropy term to the diffusion policy loss to enhance exploration. Calculating the exact entropy is intractable due to the inaccessibility of the probability density function (PDF) of diffusion policies. Therefore, we design an entropy regularization term from another perspective. Moreover, while diffusion policies offer advantages like multimodality, they also introduce large policy variance, leading to inefficient interaction with environments. To address this, we develop an efficient behavior policy through action selection to improve sample efficiency. With QVPO, diffusion policies can fully leverage their exploration capability and multimodality in the online RL paradigm. QVPO achieves state-of-the-art performance in terms of both cumulative reward and sample efficiency compared to traditional online RL methods and existing diffusion-based RL methods in MuJoCo locomotion tasks [39] .Our contributions are summarized as follows:1) Q-weighted VLO Loss. By revisiting the VLO loss of diffusion models and policy loss in online RL, we propose the Q-weighted VLO loss, which is the core component of the proposed QVPO method. We further extend its application to general RL tasks via equivalent Q-weight transformation functions. Additionally, we theoretically prove that the Q-weighted VLO loss is a tight lower bound of the policy objective in online RL.2) Diffusion Entropy Regularization & Efficient Behavior Policy. We find that the exploration ability of diffusion policies declines with limited diffusion steps. To address this, we apply a special entropy regularization term to the policy loss, which is nontrivial for diffusion policies. Additionally, to cope with the large policy variance of the diffusion model, we develop an efficient behavior policy through action selection to improve sample efficiency.3) State-of-the-art Performance. We verified the effectiveness of QVPO on MuJoCo locomotion benchmarks. Experimental results indicate that QVPO achieves state-of-the-art performance in terms of sample efficiency and episodic reward compared to both previous traditional and diffusion-based online RL algorithms. 
related works: 
 In this section, we will review the existing works that utilize diffusion models in decision-making tasks and generally divide them into four categories according to different applications.Diffusion Policies for Imitation Learning. Imitation learning is a framework for learning a behavior policy from expert datasets without explicit reward. Diffusion models can effectively fit the distribution of given datasets due to their powerful expressiveness. Some research works such as Diffusion Policy [6] , Crossway Diffusion [22] , AVDC [18] and [27] exemplify this by generating robot action sequences via diffusion based policy conditioned on visuomotor observation. Considering the reward is sparse or inaccessible in the expert dataset, BESO [32] leverages the diffusion model in the domain of goal-conditioned imitation learning to learn a goal-specified policy.Diffusion Planners. Planning in RL involves using the dynamic function to inform decision-making over a long horizon. Diffusion planners reduce compounding errors by directly constructing the complete trajectory rather than relying on one-step transitions. Diffuser [15] arranges state-action sequences into a structured two-dimensional array and samples the trajectories based on reward guidance. In subsequent work, Decision Diffuser [2] generates state sequences with classifier-free guidance and then uses inverse dynamic function to derive actions to alleviate the distribution shift problem. Latent Diffuser [20] generates sequences in a learned latent space and then reconstructs the original trajectories. A bunch of works such as [21, 3, 8] leverage hierarchical structures to enhance learning efficiency and decision-making capabilities by decomposing complex tasks into a hierarchy of sub-tasks.Diffusion Policies for Offline RL. Unlike most imitation learning problems without reward and assuming an optimal expert to provide the data, offline RL faces the challenge of learning optimal policy from suboptimal offline datasets [19] . Diffusion-QL [40] integrates the diffusion model with the conventional Q-learning framework. However, the training of Diffusion-QL is unstable in the out-of-distribution (OOD) state region, SRDP [1] reconstructs the state to alleviate the distribution shift caused by OOD. EDP [16] reduces the computation cost during diffusion inference by applying Tweedie's formula to predict actions. CEP [24] draws samples by diffusion sampling with exact guidance defined by the energy function based on contrastive learning. CPQL [5] uses a consistency model to represent the policy. Some works [12, 4, 11] utilize diffusion models to construct the optimal policy by weighted regression [29, 28] Diffusion Policies for Online RL. Online RL faces a more challenging problem in that it requires algorithms to learn the policy and make decisions in real-time interaction with the environment. DIPO [42] is the first to employ diffusion policies in online RL and proposes a novel policy improvement method. During diffusion policy updates, each action in the reply buffer is updated via action gradient to receive higher rewards. However, the action of gradient updating may deviate from the behavior policy, and the paradigm of gradient updating limits the algorithm's ability of global exploration and raises computational costs. QSM [31] introduces a new update role for diffusion policy by aligning the score of the diffusion model with the gradient of the Q-function to increase the Q-value of state-action pairs. However, it overlooks the impact of inaccuracies in the value function gradient and biases introduced during the alignment process when updating the policy. This can lead to the policy being influenced by suboptimal data within the buffer. Compared with these existing diffusion-based online RL algorithms, QVPO performs policy optimization with the Q-weighted VLO loss which can be theoretically proven to be the tight lower bound of the policy objective of online RL. This means QVPO does not incur additional errors in policy optimization. Besides, diffusion policy entropy regularization and efficient behavior policy via action selection techniques are provided to further enhance the performance of the diffusion policy in online RL. 
reinforcement learning: 
 For reinforcement learning (RL), an MDP is defined as (S, A, p, r, ρ 0 , γ), where S is the state space, A is the action space, p : S × S × A → [0, ∞) is the transition probability function of the next state s t+1 given the current state s t and the action a t , r : S × A → [r min , r max ] is the bounded reward function, ρ 0 : S → [0, ∞) is the distribution of the initial state s 0 and γ ∈ [0, 1] is the discount factor for value estimation. In an MDP, the process starts from an initial state s 0 sampled from ρ 0 and then samples actions a t from the policy π(a|s) : S × A → [0, ∞) given the state s t . Here τ = (s 0 , a 0 , s 1 , a 1 . . . ) denotes this kind of trajectory and τ ∼ π denotes the distribution of trajectory τ given the policy π(a|s). The state-value function V π (s) = E τ ∼π [ t=0 γ t r t |s 0 = s] represents the expected reward that the agent can obtain under the policy π in the state s. Compared to the state-value function, the action-state function is Q π (s, a) = E τ ∼π [ t=0 γ t r t |a 0 = a, s 0 = s] where action a 0 is provided as input. Further, the advantage function refers to the extra benefit of taking a specific action relative to taking the average action at a given state which is defined as the following:A π (s, a) = Q π (s, a) -V π (s). The goal of RL is to learn the policy that maximizes the discounted expected cumulative reward, defined as J(π) = E τ ∼π [ t=0 γ t r t ], which can be optimized via performing multiple steps of policy improvementπ k+1 = arg max π E τ ∼π k [Q π k (s, a) log(π(a | s))].Denoising diffusion probabilistic models (DDPM) [13] are powerful latent variable generative models that transform any data distribution into a simple Gaussian distribution by adding noise (forward process) and then denoise it using neural networks (reverse process). Given a dataset {x i 0 } N i=1 for x i 0 ∼ q(x 0 ), the forward process of DDPM transforms the distribution q(x 0 ) into a tractable prior Gaussian distribution by incorporating Gaussian noise in T steps with the transitions:EQUATIONwhere β t is the variance schedule. Using the Markov chain property, we can obtain the distribution of x t conditioned on x 0 :EQUATIONwhere α t = 1 -β t and ᾱt = t s=0 α s . Eventually, when T → ∞, x T converges to an isotropic Gaussian distribution. In the reverse process, DDPM first generates noise from the prior distribution p(x T ) = N (x T ; 0, I) and then gradually denoises it by learning parameterized transitions p θ (x t-1 | x t ) = N (x t-1 ; µ θ (x t , t), Σ θ (x t , t)) to fit q(x t | x t-1 , x 0 ), where θ denotes the learnable parameters. The training objective of DDPM is to maximize the Variational LOwer Bound (VLO) defined as L VLO = E q(x 0:T ) log p θ (x 0:T ) q(x 1:T |x0) . Finally, with ϵ ∼ N (0, I), the loss in DDPM takes the form of:EQUATION)Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes]• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
q-weighted variational policy optimization: 
 In this section, we will first revisit the VLO loss [13] of the diffusion model and the policy objective of online RL [37] . Then, we derive the Q-weighted VLO loss, which is the principal component of the proposed Q-weighted Variational Policy Optimization (QVPO). However, the Q-weighted VLO loss cannot be directly applied in general RL tasks. To address this issue, we also provide equivalent Q-weight transformation functions. Since the derived Q-weighted VLO loss is the tight lower bound of the policy loss in online RL, QVPO can effectively avoid the drawbacks of existing online RL methods for diffusion policies [42, 31] mentioned in Section 2.QVPO also involves practical techniques for the underlying issues of optimizing diffusion policies. Firstly, to further enhance the exploration ability of diffusion policy, we develop a special entropy regularization. This is not trivial for diffusion policy since the log probability of the state-action pair is not available. Besides, we devise an efficient behavior policy via action selection for the diffusion model to avoid the large policy variance of diffusion, which results in sample inefficiency. With these practical techniques, diffusion policy can achieve better performance with fewer online interactions with the environments. Finally, the training procedure of QVPO is shown in Figure 1 .As mentioned in DIPO [42] , optimizing diffusion policies in the online RL paradigm is nontrivial, which principally results from two reasons. In one aspect, if we directly apply the deterministic policy gradient to diffusion policies like Diffusion-QL [40] , the backpropagation chain through the denoising procedure of the diffusion model becomes quite long. This leads to high computational costs and instability during training, which severely restricts the performance of diffusion policies in online RL. In another aspect, samples from the optimal policy are required when directly using the variational bound objective to train diffusion policies. However, these optimal samples are typically unavailable in online RL.In this context, we revisited the VLO objective of diffusion models and the RL policy objective. To our surprise, we discovered that by adding the appropriate weights to the VLO objective, it becomes a tight lower bound of the RL policy objective under certain conditions. Then, these action samples will be selected and endowed with different weights according to the Q value given by the value network. Besides, action samples from uniform distribution are also created for the diffusion entropy regularization term. With these action samples and weights, we can finally optimize the diffusion policy via the combined objective of Q-weighted VLO loss and diffusion entropy regularization term.is the tight lower bound of the objective of RL policyE s,a∼p(s ′ |s,a),π k (a|s) [Q(s, a) log(π θ (a|s))] ,and the equality holds when the policy converges.The proof can be referred to in the supplementary materials. According to this theorem, we can derive the Q-weighted variational bound loss as (4), which can be applied to diffusion policy optimization.EQUATION)According to [13] , removing the coefficientEQUATIONt αt(1-ᾱt) does not affect the training of diffusion. Hence, the final Q-weighted VLO loss is defined asEQUATIONWhile we can now use (5) to optimize the diffusion policy, two issues remain to be resolved.1) Negative Q value. In real-world decision-making tasks, it is difficult to ensure that the returned reward is always non-negative, which means the Q value may be negative for some state-action pairs. Therefore, to apply QVPO in practical tasks, we must address situations where the Q(s, a) value is negative for certain states and actions.2) High-quality Training Samples. According to (5) , achieving significant policy improvement requires obtaining certain rare state-action samples with high Q values. However, this is a significant challenge with limited interaction with the environment. This issue is also common in other online RL methods based on weighted policy training, such as RWR [30] , and it hinders their application to real-world tasks.To resolve these two problems in training diffusion policies with the Q-weighted VLO loss, we propose equivalent Q-weight transformation functions to convert the Q value into an equivalent positive Q weight and leverage the powerful data-synthesizing ability of the diffusion model to generate high-quality samples for training. The new equivalent Q-weighted VLO loss is defined in (6) . We will introduce the detailed implementation of these solutions in the following contexts.EQUATION)Theorem 2. (Optimal Solution in One Policy Improvement Step) Considering the optimization problem in policy improvement stepπ k+1 = arg max π E τ ∼π k [Q π k (s, a) log(π(a | s))],the optimal policy in a given state s isEQUATIONwhen there exists a such that Q(s, a) > 0, andEQUATIONwhen Q(s, a) ≤ 0 for any a ∈ A, δ(x -a) is the Dirac function, N is the cardinality of set {a | Q(s, a) = max a Q(s, a), π(a | s) > 0}, π(a | s) is the behavior policy, p(a | s)is the policy to be optimized and I Q(s,a)>0 is the indicator function that judges whether Q(s, a) > 0.The proof can be referred to in the supplementary materials. Hence, with Theorem 2, we can obtain the probability density function (PDF) of the optimal policy in one policy improvement step and further use p(a|s) π(a|s) as the weight to achieve an equivalent optimized diffusion policy according to (6) . However, the difficulty of judging whether any Q(s, a) ≤ 0 in a certain state s is still a problem. Therefore, we first come up with an approximate solution "qcut" weight transformation function, which returns Q(s, a) when Q(s, a) ≥ 0 and a small value ϵ > 0 when Q(s, a) < 0. Furthermore, the "qcut" weight transformation function just endows the best action with the weight mentioned above and discards the left action samples via setting them as zero.In short, qcut weight transformation function takes the formulation of the above two situations into account and "cuts" them via a small value ϵ > 0. Notably, we remove the denominatorI Q(s,a)>0 π(a | s)Q(s, a)da in qcut weight transformation function. This is because the denominator I Q(s,a)>0 (a)π(a | s)Q(s, a)da is somewhat like the V (s), which denotes the "importance" of the state s. Therefore, removing this item can allow important states to obtain larger weights during training, which is confirmed in practice.However, we find that the qcut weight transformation function does not work very well when Q values in most state-action pairs are less than zero. To address this issue, we propose another "qadv" weight transformation function, which is much simpler but performs better in our experiments.EQUATIONwhere A(s, a) = Q(s, a) -V (s) is the advantage function. qadv weight transformation function replaces the Q(s, a) with the advantage A(s, a) to avoid the second situation. Moreover, applying Q value and advantage in policy optimization has been proved equivalent and can also reduce the training variance in previous works like [25] .It is worth noting that, for qadv weight transformation function, we do not require an extra network to approximate V (s) to calculate A(s, a). We can always sample a certain number of actions for a state s with diffusion policy and estimate V (s) using the average of their Q(s, a). Besides, for qadv function, we also recommend just selecting the best action sample with the largest A value for each state in policy optimization, which can further improve the quality of training samples and reduce the training cost.Remark. Although the weighted VLO loss of QVPO looks similar to AWR [28] and EDP [16] , they are different intrinsically. With the Q-weighted VLO loss, QVPO trains the diffusion policy with the weighted sample from the current policy, while AWR or EDP trains the diffusion policy with the weighted sample from the policy in the offline dataset (replay buffer). These "on-policy" samples truly benefit the online training. Moreover, AWR and EDP utilize the exp weight function; but this function is too conservative for online RL.While diffusion policies can achieve impressive performance with the Q-weighted VLO loss in online RL, their powerful exploration capabilities have yet to be fully harnessed. As noted in Diffusion-QL [40], the policy expressiveness of the diffusion model decreases with fewer diffusion steps. In our experiments, we find that not only does policy expressiveness decline, but the exploration capability of the diffusion model also diminishes when the number of diffusion steps is reduced. However, it is essential to limit the number of diffusion steps to avoid excessive training and evaluation costs in real-world applications. Therefore, studying how to enhance the exploration ability of the diffusion model with a limited number of diffusion steps is necessary.Adding an extra entropy regularization term to the policy loss appears to be a good solution, as it has been validated for categorical policies in discrete action spaces and Gaussian policies in continuous action spaces. However, estimating the entropy for a diffusion policy is nontrivial due to the inaccessibility of the log-likelihood of action samples. Moreover, maximizing the entropy of a policy can be viewed as narrowing the distance between the policy and a maximum entropy distribution (i.e., uniform distribution) in some sense. Thus, by recalling the VLO loss of the diffusion model, the entropy of a diffusion model can be increased with training samples from the uniform distribution. Based on this idea, we propose an entropy regularization term for diffusion policies as follows,EQUATION)where ω ent (s) = ω ent N i=1 ωeq(s,ai) Nis the coefficient related to state s for balancing the trade-off between exploitation and exploration. Here, N represents the number of selected training samples from the diffusion policy for state s. When ω ent is large, the exploration ability of the diffusion policy is improved, whereas the exploitation ability is reduced. Figure 2 is the experiment results on a continuous bandit toy example that clearly illustrates the effect of this entropy regularization term on the diffusion policy, and the concrete reward function of this bandit problem is R(x) = 3 i=1 w i 1 2πσ 2 i exp -1 2σi (x -µ i ) T (x -µ i ), where w i = 1.5, σ i = 0.1, and µ i = [-1.35, 0.65] T , [-0.65, 1.35] T and [-1.61, 1.61] T respectively.Despite the diffusion model that allows the online RL agent to seek better policies, it also introduces a large policy variance. This results in inefficiency for online interactions of the behavior policy with the environment. To address this issue, we propose an efficient behavior policy via action selection for diffusion model π K θ (a | s), which improves the sample efficiency. Specifically, it is motivated by the idea that the efficiency of action samples from behavior policy depends on their Q values. In other words, action samples from behavior policy with high Q values indicate that the following trajectory also has a large underlying reward and is of great significance for training. Specifically, the efficient behavior policy π K θ (a | s) is defined as:π K θ (a | s) ≜ argmax a∈{a1,••• ,a K ∼π θ (a|s)} Q(s, a).Furthermore, while using the efficient behavior policy π K θ (a | s) as a behavior policy can improve sample efficiency, it is not recommended to apply the same action selection number K to the target policy for calculating the TD target. This is due to the large policy variance of the diffusion policy, which can result in a serious overestimation of the target Q value. Hence, if we choose the action selection number K b for the behavior policy, a smaller action selection number K t < K b for the target policy is often a good choice in practice.  
experiments: 
 In this section, we first describe a practical implementation of QVPO, as shown in Algorithm 1, which is implemented in an off-policy fashion. Then, we evaluate the proposed algorithm on different decision-making tasks and with various hyper-parameters. With these experimental results, we aim to answer three questions:• How does QVPO compare to previous popular online RL algorithms and existing diffusionbased online RL algorithms? • How does the entropy regularization term affect the performance of QVPO? • Can the K-efficient behavior policy significantly improve the sample efficiency of QVPO? Sample the action using the diffusion policy π K b θ (a | s t ).Take the action a t in the environment and store the returned transition in D.Sample a mini-batch B of transitions in D.Generate N d samples from π θ (a | s), and N e samples from U(a, a) for each state s in B.Endow the N d samples with weights (9).Select an action sample a max with maximum weight among N d samples for training.Endow the N e samples with the weight ω ent (s) = ω ent • ω eq (s, a max ).Update the parameters of the diffusion policy using the summation of ( 6) and (10) .10:Construct TD target as y t = r t + γQ ω (s t+1 , π Kt θ (a | s t+1 )) for each (s t , a t , r t , s t+1 ) in B.Update the parameters of the value network using MSE loss. 12: end forTo demonstrate the effectiveness of our method, we compared QVPO with six other online model-free RL algorithms: two off-policy algorithms (TD3 [9] and SAC [10] ), two on-policy algorithms (PPO [34] and SPO [3] ), and one advanced diffusion-based online RL algorithm (DIPO [42] , QSM [31] ). These comparisons were conducted on five MuJoCo locomotion tasks [39] . we plotted the learning curves of QVPO and the six other algorithms over five random runs, as shown in Figure 3 . The solid curve represents the average return, and the transparent shaded region represents the standard deviation. Each experiment was conducted over 1e6 training epochs. According to the learning curves, QVPO achieves state-of-the-art performance compared to the other six algorithms, and converges much faster than the other algorithms, further demonstrating its sample efficiency. Additionally, the practical implementation of QVPO follows SAC in critic part, which utilizes doubled Q networks and only use the minimum for policy and critic update. stage, we apply the efficient behavior policy with a large action selection number K = 32 instead of the deterministic denoising procedure used in DIPO [42] . As illustrated in [41] , the high-probability region in each diffusion step is not the origin but a Gaussian sphere. Hence, using the deterministic denoising procedure in the evaluation stage is not a good choice. In that case, using the proposed efficient behavior policy during evaluation is more reasonable. More details related to the experiments can be found in the supplementary materials. To further examine the significance of each component in QVPO, we also conducted the ablation study and parameter analysis on the effects of the entropy regularization and the K-efficient behavior policy. Here We use the experiment on Ant-v3 as an example since the final results are similar across different locomotion tasks. 4 , without the diffusion entropy regularization loss, QVPO with 100 diffusion steps achieves much better performance than QVPO with 20 diffusion steps. However, when the entropy regularization loss is applied, QVPO with 20 diffusion steps achieves close performance to QVPO with 100 diffusion steps, which verifies the effectiveness of the entropy regularization term. Kb=1 & Kt=1 Kb = 4 & Kt=1 Kb=4 & Kt=4 Kb=4 & Kt=2 Kb = 20 & Kt=2 
conclusion: 
 In this paper, we proposed a novel diffusion-based online RL algorithm called QVPO that sufficiently exploits the expressiveness and multimodality of diffusion policy. QVPO leverages the Q-weighted VLO loss, a core component that serves as a tight lower bound of the policy objective in online RL under certain conditions, facilitated by Q-weight transformation functions. Additionally, we designed a special entropy regularization term to enhance the exploration capabilities of diffusion policies and the efficient behavior policy to improve sample efficiency by reducing the behavior policy variance. Our comprehensive experiments on MuJoCo continuous control benchmarks demonstrate that QVPO achieves state-of-the-art performance in terms of both cumulative reward and sample efficiency, surpassing both traditional and existing diffusion-based online RL methods.However, there are still several underlying challenges in applying diffusion policies to online RL that require further exploration. For instance, although we developed a special entropy regularization loss to approximate the effect of maximizing entropy, it lacks the ability to adaptively adjust the entropy term and cannot incorporate the entropy of the diffusion policy into the TD target for soft policy iteration, as seen in SAC. Future work will focus on developing adaptive entropy adjustment mechanisms and integrating entropy into the TD target to enable soft policy iteration, which we believe will further enhance the performance of diffusion policies in online RL. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: Our paper primarily focuses on theoretical research in how to employ the diffusion model in online reinforcement learning, with no consideration of societal impacts.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our experiment of this paper was conducted on five MuJoCo locomotion tasks as Hopper-v3, Walker2d-v3, Ant-v3, HalfCheetah-v3, and Humanoid-v3, which poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The implementation of SAC, TD3, PPO, SPO, and DIPO are cited properly in Appendix B. Other assets are not applied here.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: A.1 Proof of Theorem 1 Theorem 1. (Lower Bound of RL Policy Objective) If Q(s, a) ≥ 0 for any state-action pair (s, a), the Q-weighted variational bound objective of diffusion policyE s,a∼p(s ′ |s,a),π k (a|s) Q(s, a 0 ) • E a 1:T ∼q(a 1:T |s,a0) log π θ (a 0:T | s) q (a 1:T | s, a 0 )is the tight lower bound of the objective of RL policyE s,a∼p(s ′ |s,a),π k (a|s) [Q(s, a) log(π θ (a|s))] ,and equality holds when the policy converges.Proof. The policy objective of online RL ismax θ E s,a∼p(s ′ |s,a),π k (a|s) [Q(s, a) log(π θ (a|s))] ,where π k (a | s) indicates the behavior policy in state s, π θ (a | s) is the policy to be optimized, and Q(s, a) represents the Q value of the state-action pair (s, a). Then, if π θ (a | s) is a diffusion policy, the inner-term Q(s, a) log (π θ (a|s)) equals to Q(s, a 0 )E a 1:T ∼q(a 1:T |s,a0) log π θ (a 0:T | s) q (a 1:T | s, a 0 ) +Q(s, a 0 ) D KL (π θ (a 1:T | s, a 0 ) ∥q (a 1:T | s, a 0 )) .According to the Jensen's Inequality and Q(s, a) ≥ 0, the second KL divergence termQ(s, a 0 ) D KL (π θ (a 1:T | s, a 0 ) ∥q (a 1:T | s, a 0 )) = Q(s, a 0 )E a 1:T ∼q(a 1:T |s,a0) log π θ (a 1:T | s, a 0 ) q (a 1:T | s, a 0 ) ≥ Q(s, a 0 ) log E a 1:T ∼q(a 1:T |s,a0) π θ (a 1:T | s, a 0 ) q (a 1:T | s, a 0 ) = Q(s, a 0 ) log(1) = 0.Obviously, the equality holds when π θ (a 1:T | s, a 0 ) = q (a 1:T | s, a 0 ). Finally, the Q-weighted variational bound objectiveE s,a0∼p(s ′ |s,a0),π k (a|s) Q(s, a 0 )E a 1:T ∼q(a 1:T |s,a0) log π θ (a 0:T | s) q (a 1:T | s, a 0 )is the tight lower bound of the RL policy objectiveE s,a∼p(s ′ |s,a),π k (a|s) [Q(s, a) log(π θ (a|s))]A.2 Deriviation of Q-weighted VLO Loss For each state s, we haveE a 1:T ∼q(a 1:T |s,a0) log q (a 1:T | a 0 ) π θ (a 0:T ) = E a 1:T ∼q(a 1:T |s,a0) log T t=1 q (a t | a t-1 ) π θ (a T ) T t=1 π θ (a t-1 | a t ) = E a 1:T ∼q(a 1:T |s,a0) -log π θ (a T ) + log T t=1 q (a t | a t-1 ) T t=1 π θ (a t-1 | a t ) = E a 1:T ∼q(a 1:T |s,a0) -log π θ (a T ) + T t=1 log q (a t | a t-1 ) π θ (a t-1 | a t ) = E a 1:T ∼q(a 1:T |s,a0) -log π θ (a T ) + T t=2 log q (a t | a t-1 ) π θ (a t-1 | a t ) + log q (a 1 | a 0 ) π θ (a 0 | a 1 ) = E a 1:T ∼q(a 1:T |s,a0) -log π θ (a T ) + T t=2 log q (a t-1 | a t , a 0 ) q (a t | a 0 ) π θ (a t-1 | a 0 ) q (a t-1 | a 0 ) + log q (a 1 | a 0 ) π θ (a 0 | a 1 ) = E a 1:T ∼q(a 1:T |s,a0) -log π θ (a T ) + T t=2 log q (a t-1 | a t , a 0 ) π θ (a t-1 | a t ) • q (a t | a 0 ) q (a t-1 | a 0 ) + log q (a 1 | a 0 ) π θ (a 0 | a 1 ) = E a 1:T ∼q(a 1:T |s,a0) -log π θ (a T ) + T t=2 log q (a t-1 | a t , a 0 ) π θ (a t-1 | a t ) + log q (a t | a 0 ) q (a t-1 | a 0 ) + log q (a 1 | a 0 ) π θ (a 0 | a 1 ) = E a 1:T ∼q(a 1:T |s,a0) -log π θ (a T ) + T t=2 log q (a t-1 | a t , a 0 ) π θ (a t-1 | a t ) + log q (a T | a 0 ) q (a 1 | a 0 ) + log q (a 1 | a 0 ) π θ (a 0 | a 1 ) = E a 1:T ∼q(a 1:T |s,a0) -log π θ (a T ) + T t=2 log q (a t-1 | a t , a 0 ) π θ (a t-1 | a t ) + log q (a T | a 0 ) -log q (a 1 | a 0 ) + log q (a 1 | a 0 ) -log π θ (a 0 | a 1 ) = E a 1:T ∼q(a 1:T |s,a0) log q (a T | a 0 ) π θ (a T ) + T t=2 log q (a t-1 | a t , a 0 ) π θ (a t-1 | a t ) -log π θ (a 0 | a 1 ) = E a 1:T ∼q(a 1:T |s,a0) D KL (q (a T | a 0 ) ∥π θ (a T )) + T t=2 D KL (q (a t-1 | a t , a 0 ) ∥π θ (a t-1 | a t )) -log π θ (a 0 | a 1 ) = E a0,ϵ β 2 t 2σ 2 t α t (1 -ᾱt ) ϵ -ϵ θ √ ᾱt a 0 + √ 1 -ᾱt ϵ, s, t 2 .Thus, the Q-weighted VLO loss can be derived asmax θ E s,a0∼π k (a|s) Q(s, a 0 )E a 1:T ∼q(a 1:T |s,a0) log π θ (a 0:T | s) q (a 1:T | s, a 0 ) ≜ min θ E s,a0∼π k (a|s) Q(s, a 0 )E a 1:T ∼q(a 1:T |s,a0) log q (a 1:T | s, a 0 ) π θ (a 0:T | s) ≜ min θ E s,a∼π k (a|s),ϵ,t β 2 t 2σ 2 t α t (1 -ᾱt ) Q(s, a) • ϵ -ϵ θ √ ᾱt a + √ 1 -ᾱt ϵ, s, t 2 .A. 3when there exists a such that Q(s, a) > 0, andEQUATIONwhen Q(s, a) ≤ 0 for any a ∈ A.Assume the new diffusion policy after one QVPO iteration can be approximately expressed asπ new (a | s) ≈ (1 -p data (s)A π old (s, a ⋆ )η)π old (a | s) + p data (s)A π old (s, a ⋆ )η I a∈N (a ⋆ |s,ϵ) (a) S N (a ⋆ |s,ϵ)where a ⋆ | s is the action that can maximize Q π old (s, a) in the state s, N (a ⋆ | s, ϵ) is the neighborhood of a ⋆ | s with a small radius ϵ, S N (a ⋆ |s,ϵ) is the area of this neighborhood, and p data (s) is the sampling distribution of the state. This assumption is straightforward: the training sample's generation probability in the diffusion model will be improved and the improved probability is proportional to its weight. Now consider the improvement of the RL objective:J (π new ) -J (π old ) = E s∼ρ0 [V πnew (s) -V π old (s)] = E s∼ρ0 E a∼πnew(a|s) [Q πnew (s, a)] -V π old (s) = E s∼ρ0 E a∼πnew(a|s) [Q πnew (s, a) -Q π old (s, a)] + E s∼ρ0 E a∼πnew(a|s) [Q π old (s, a)] -V π old (s) = E s∼ρ0 E a∼πnew(a|s) [Q πnew (s, a) -Q π old (s, a)] + E s,a∼ρ0,πnew(a|s) [A π old (s, a)]The first term here can be further expanded according to the Bellman equation:E s∼ρ0 E a∼πnew(a|s) [Q πnew (s, a) -Q π old (s, a)] = γE s∼d 1 πnew [V πnew (s) -V π old (s)] where d 1πnew denotes the probability distribution of state in time step t = 1 with policy π new . Repeating the above operation, we will obtain:J (π new ) -J (π old ) = ∞ t=0 γ t E s,a∼d t πnew ,πnew [A π old (s, a)] = 1 1 -γ E s∼dπ new E a∼πnew(•|s) [A π old (s, a)] ≈ 1 1 -γ E s∼dπ new (1 -p data (s)A π old (s, a ⋆ )η)E a∼π old (•|s) [A π old (s, a)] + p data (s)A 2 π old (s, a ⋆ )η I a∈N (a ⋆ |s,ϵ) (a) S N (a ⋆ |s,ϵ) = 1 1 -γ E s∼dπ new p data (s)A 2 π old (s, a ⋆ )η I a∈N (a ⋆ |s,ϵ) (a) S N (a ⋆ |s,ϵ) ≥ 0Moreover, although QVPO is much slower than classical RL methods like SAC in inference, the inference time (6ms) still is acceptable. To our knowledge, most existing real robots only require 50-Hz control policy (i.e. output action per 20 ms). Besides, just like QSM, the inference time of QVPO can be further improved with jax framework if it is necessary. Hence, the inference time is not a bottleneck to applying QVPO to real applications.NeurIPS Paper Checklist Justification: The analysis of limitations is provided in Section 6.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 


Decision: The reviewers appreciate the novelty of the work as one of the first results applying diffusion policies to online RL and obtaining reasonable results. However, they were all concerned about some limitations in the experiments and several claims made in the paper. Some of these concerns were addressed during the rebuttals and as a result reviewers raised their scores. However, it is still a borderline paper, slightly above the bar. I believe the novelty of the approach could be a good reason for acceptance but I strongly recommend that the authors take the questions/concerns raised by the reviewers into account in preparing the next revision of their work. Especially, it would be great if they can run some experiments in more difficult environments.

Reviews: review: paper_summary: paper_summary: The method proposes using Q-weight diffusion loss to train agents in online RL. Instead of approximating log probability, the paper uses advantage weight tuning to maintain the entropy of the actor.
strength_weakness: strength_weakness: My main concern is that the paper has too much heuristic design regarding equivalent Q-weight transformation functions, the entropy regularization term, and Action Selection. Especially, Eq(9) and Eq(10) are called equivalent Q-weight transformation functions, but they are not mathematically equivalent to Eq(5).
questions: questions: 1. Equivalent Q-weight transformation functions are used to solve the negative Q value problem. Have the authors considered tuning rewards (for example, adding a fixed constant to all rewards to make all rewards positive) to guarantee Q values are all positive? It may be the most straightforward way to avoid the negative Q value issue.

2. There is a typo in line 208 where the right parenthesis should be "]".

3. For Eq(9), how do you estimate $\max_a Q(s,a)$ in practice?

4. Even though critic training is not the main point of the paper, I suggest adding a section in the Appendix to explicitly describe it for completeness.

5. In Figure 5, some $K_b,K_t$ combinations may lead to extreme training collapse. Did the authors experiment with higher values of $K_b,K_t$ while maintaining $K_t<K_b$?
limitations: limitations: My main concern is that the design of equivalent Q-weight transformation functions is heuristic and may need some theoretical intuition or backup.
paper_summary: paper_summary: The paper uses a diffusion policy to address online RL. The method works by weighting the diffusion model VLO loss using advantages computed with a Q function. Additionally, they add an entropy maximization term to aid with exploration. The authors present results indicating strong performance on a variety of continuous control tasks as well as ablations verifying the necessity of several components of their method.
strength_weakness: strength_weakness: There are several claims made in the paper that I don't believe are substantiated. Some important ones:
 - In the introduction and RW section you make claims about the DIPO and QSM baselines that don't seem to be proven anywhere
 - You take credit for several things that aren't yours. Your "efficient diffusion policy" trick is used extensively in other work and the weighted VLO loss is one of the methods presented in EDP (Kang et al.)

I think there need to be some changes to the experiments.
 - Most important: the comparison to other RL algorithms is unfair since you tune your algorithm per environment whereas other algorithms have a fixed set of hyperparameters. You need to either tune all relevant hyperparameters for all other algorithms for all environments or fix yours.
 - One of your key claims is that the diffusion policy is a good fit for RL because of its ability to model multimodal action distributions and yet you don't consider any environments where this is the case. Rather, you simply demonstrate improved sample efficiency on a handful of tasks that clearly can be solved using a MLP policy parameterization
 - You really should compare to QSM
 - The ablation studies are a bit incomplete in my opinion. It would be good to see results from all five environments. Also, it would be good to run an ablation where you omit the policy loss weighting, leading to an algorithm similar to IDQL, to verify that the policy weighting is important and that your policy isn't relying on the efficient diffusion policy step. I assume this ablation will go well for you based on the fact that the $K_b=1$ ablation still learns a policy but this will strengthen the paper.
 - You mention qcut, then say it doesn't work and never mention it again. Either show an experiment proving that or remove it from the paper
 - I believe that slow wall clock time, either for training or online rollouts, is potentially a big limitation of the method. Wall clock time is vaguely mentioned in the appendix but it would be good to see some data about this in the paper
questions: questions: Random questions and comments
 - The second and third sentence of the abstract seem to contradict one another. How is it that diffusion policies can improve exploration (which is only relevant to online RL) if they have only really been applied to offline RL?
 - In the introduction you claim that DIPO has issues with limited exploration but I don't see any proof for that claim in your or their paper
 - In the introduction you claim that QSM has issues that prevent it from converging to optimal policies that your method does not have, but you never compare to QSM to demonstrate this win. I think QSM is an important comparison that really should be added to the paper since it is probably the closest to your method.
 - I don't understand the purpose of including qcut in the manuscript since you immediately propose a different approximation that works better and then never mention qcut again.
 - I'm very confused by Figure 2. How did you generate this? What does 'explorable area' mean? 
 - Section 4.4 seems to imply that selecting an action by sampling several and then choosing the one that maximizes the Q value is a novel contribution, but this has been done on several occasions including by Diffusion QL, Efficient Diffusion Policy, Implicit Diffusion Q Learning, etc. This should be rewritten and these works should be cited. Also it is frequently referred to as "efficient diffusion policy" in the paper which is confusing since there is another paper (Kang et al.) that is called the same thing.
 - The policy improvement objective in this paper is actually very similar to one of the variants of EDP (Kang et al.), specifically shown in Equation 13 of https://arxiv.org/pdf/2305.20081. This paper still has reasonable novelty compared to that paper but that similarity should be better acknowledged.
 - Figure 4 is a nice result but it would be helpful to also see steps=100 & ent.
 - I'm curious about the wall clock training time of QVPO compared with baselines. Does everything take about 10 hours? I would expect QVPO to be quite a bit slower.
 - Is $\omega_\textrm{ent}$ difficult to tune?



Nit picks (not penalizing you for these but thought I should point them out):
 - The text in the plot legend is too small and the plots would be easier to read if the authors applied some smoothing
 - The transitions used at the beginning of the sentences in lines 163-170 (besides, firstly, notably, etc) are very distracting and the paragraph would sound better if you removed all of them.
 - There is an extra " on line 237 for some reason
 - Appendix B: hyperparameters should not be hyphenated
 - The legends of the plots look sloppy. Matplotlib allows the use of LaTeX and you should make use of that
limitations: limitations: The authors mention one limitation--the lack of adaptive entropy regularization. However, they miss other potential limitations, such as slower wall clock training time or the inability to output extended chunks of actions, as is common for diffusion policies applied to BC.
paper_summary: paper_summary: The paper proposes a novel model-free online reinforcement learning (RL) algorithm called Q-weighted Variational Policy Optimization (QVPO), which leverages the expressiveness and multimodality of diffusion models. By introducing Q-weighted variational loss and entropy regularization, the authors aim to overcome the limitations of unimodal policies and enhance exploration capabilities. The algorithm is validated through experiments on MuJoCo continuous control benchmarks, demonstrating state-of-the-art performance in terms of both cumulative reward and sample efficiency.
strength_weakness: strength_weakness: 1. Experiment Scope: The experiments are limited to D4RL datasets, which do not cover the full spectrum of possible RL environments. The generalization of the proposed method to other types of tasks, such as discrete action spaces or tasks with sparse rewards, is not demonstrated, eg. Adroit, Humanoid, MetaWorld, Maniskill, KUKA Pick-and-Place.
2. Ablation Studies: While the paper includes ablation studies, they are not exhaustive. Additional ablation studies are needed to isolate the contributions of individual components, such as the Q-weighted variational loss and the entropy regularization term. It is unclear how much each component independently contributes to the overall performance improvements.
3. Diffusion Policy Variance Reduction: The proposed method for reducing diffusion policy variance via action selection aims to improve sample efficiency. However, this approach may sacrifice the exploration capabilities of the diffusion policy, thus limiting its multimodal potential. This trade-off is not discussed in the paper, and no ablation study is provided to evaluate the impact.
questions: questions: 1. Can the proposed algorithm be adapted for tasks with discrete action spaces or sparse rewards, and if so, how?
2. How is the diffusion policy initialized, and what impact does this initialization have on the performance of the algorithm? If the policy is tranined from scratch, how is the Q-function trained at the beginning stage?
3. The efficient diffusion policy method reduces policy variance but may limit exploration. How do you balance exploration and exploitation, and can you provide an ablation study to verify the effects of this trade-off?
4. From Figure 3, QVPO appears to converge faster. Can you provide an explanation or experiments to support this observation?
5. How does the computational complexity of QVPO compare with existing diffusion-based online RL methods in terms of training time and resource utilization?
limitations: limitations: The authors have addressed the limitations of the proposed method.
paper_summary: paper_summary: This paper introduces QVPO, a new model-free algorithm that trains a diffusion policy online. It proposes a Q-weighted VLO loss by weighting the original diffusion model objective with Q-values. To address the issue of negative Q-values, the algorithm uses advantage instead. QVPO also encourages exploration by mimicking the uniform distribution and reduces inference variance by executing actions with higher Q-values.
strength_weakness: strength_weakness: - Several claims require additional support. For instance, while Figure 2 is informative in demonstrating the ideal effect of the entropy term, experimental results on toy examples would provide stronger evidence than an illustration.

- Since QVPO relies on action samples for both optimization and action selection, this will increase computational demands.

- The tasks are limited and relatively trivial, making it difficult to tell performance in more complex tasks, such as robot manipulation. In particular, when the action space is larger, it is unclear how the entropy term will scale given its reliance on uniform action distributions.
questions: questions: - Can the author provide an explanation for why the exploration ability of diffusion policies declines with a limited number of diffusion steps? Additionally, what happens when the diffusion steps are even fewer, say 5, a common value in other diffusion policy literature? I suppose the entropy term should perform better in such cases.

- If the maximum entropy term is used throughout the training, the policy may continue to follow the uniform distribution in the later stages, potentially hurting performance and convergence. If this is the case, would a scheduling mechanism improve the final performance?

- Can the author explain why the Q-weighted VLO loss is superior to QSM regarding the additional errors in policy optimization?

- In line 300, it is mentioned that action selection is used during inference, which is a standard technique in offline RL. However, is the same technique applied to the baselines? If not, this would be unfair.
limitations: limitations: --

Comments: title: Response to rebuttal
Please let us know if any further questions! Thanks so lot!
Clarification on the meaning of training epoch and the impact of QVPO -- Reply to Reviewer Kypj
Illustrations about the performance of our baselines ---- Reply to Reviewer Kypj
Most comments addressed but I'm worried about performance of all your baselines.
comment: Thanks for your detailed rebuttal. The authors have addressed part of my concerns. I think the paper is slightly beyond the accept line. I will keep my score. Thank you.
**Q11**: What if the rewards for each center are unbalanced and have different weights?

**A11**: Thanks for raising the score. That is a good question. Actually, We conducted the experiments under this setting before. We find that QVPO with the entropy term finally converges to the optimal center under an unbalanced center setting, while QVPO without the entropy term has a certain probability of falling into the sub-optimal center, especially in the cases where the imbalance between different modes is not obvious. Concretely, QVPO with entropy term usually first explores these three centers (show its multimodality) and then converges to the optimal one. The convergence rate depends on the imbalance degree (i.e., the greater the relative weight difference among the three modes, the faster the convergence).
Thanks for the response. Regarding A8, I am curious what if the rewards for each center are unbalanced and have different weights? Overall, I think the rebuttal alleviates my concerns and I will raise my score to 5 (slightly beyond the accept line).
Thanks for your comments and questions as well. 

**Q8**: Can the authors elaborate more on how the continuous bandit experiment is actually being done, e.g., how do you sample the initial points?

**A8**: The continuous bandit experiment is done similarly to Algorithm 1. Concretely, we sample 64 actions from the current diffusion policy and select the best according to the reward function $Q(x)=\sum_{i=1}^3w_i\frac1{2\pi\sigma_i^2}\exp\left(-\frac1{2\sigma_i}(x-\mu_i)^T(x-\mu_i)\right)$, where $w_i=1.5$, $\sigma_i=0.1$, and $\mu_i=[-1.35,0.65]^T$, $[-0.65,1.35]^T$ and $[-1.61,1.61]^T$ respectively. Then, we train the diffusion policy with the best action sample (with & without entropy term) in each training epoch. We plot the evaluation results per 10 training epochs with enough samples from the diffusion policy.

The initial points (we guess you mean the first column of Figure 1 in the attached pdf file) are sampled from an initial diffusion policy pre-trained from a Gaussian distribution $\mathcal{N}(0, 0.25)$. Notably, **this pre-training procedure is not required in practical implementation**. The pre-training procedure from a given Gaussian distribution (not good) actually increases the convergence difficulty for QVPO. We use this pre-training procedure just for better visualization; otherwise, QVPO will converge very fast for such a toy example and thus cannot show a good visualization. 

**Q9**: Do you use any parallel environment during training? The multiple sampling and selection procedure of QVPO may become a bottleneck, let alone GPU simulators with thousands of parallel environments.

**A9**: No, we do not use any parallel environment during training for fair comparison. Firstly, we need to clarify that the parallelism of QVPO is not in **the online environment interaction stage** but **in the policy optimization stage**. As mentioned in lines 178-179, the difficulty of diffusion policy optimization is how to obtain the optimal action samples. Previous diffusion-based RL method like DIPO finds the optimal action sample by performing multiple gradient updates on the actions in the replay buffer. However, the multiple gradient updates in DIPO cannot be implemented in parallel. In contrast, QVPO finds the optimal action samples via multiple sampling and selection, which can be parallelly executed. In that case, QVPO can speed up policy optimization compared to DIPO. 

Besides, although the idea of action selection is also applied to online environment interaction (i.e., "efficient diffusion policy" in Section 4.4), we believe this trick of QVPO is not a bottleneck with parallel environment interaction. This is because the action selection number in "efficient diffusion policy" is commonly chosen to be very small ($K_b=4$ is enough generally). In that case, it will not affect the training time too much with parallel environments. In fact, the GPU utilization and memory consumption of QVPO is limited (i.e., 55% & 598MB) even in the policy optimization stage, where 256 states are given parallelly and we need to sample 64 actions for each of them. 

**Q10**: The authors claim that QVPO can **leverage** the multimodality of diffusion policies (line 60), however, the authors didn't prove it or even mention it in the rest of the paper. I feel like this claim is very strong and you should verify it in experiments.

**A10**: We have no intention to show the multimodality of diffusion policy (which has been proven in DIPO, QSM). Instead, we want to efficiently **leverage** the multimodality of diffusion policy to achieve better performance in continuous control tasks. The multimodality is generally hard to visualize, especially for complex real tasks. Therefore, we develop a toy example to show this property. As shown in Figure 1 of the attached file, the proposed diffusion entropy term and the sampling-based policy optimization make it easy for QVPO to explore broader action space (Figure 1) and thus can verify that QVPO can better leverage multimodality.
**Q8**: You mentioned that "QVPO can efficiently leverage GPU parallel computing, and the multiple-sampling and selecting procedure does not significantly impact time compared to gradient-based optimization like DIPO." Could you clarify the parallel settings used to evaluate the training and inference times for your method?

**A8**: Thanks for your comments as well. As mentioned in lines 178-179, the difficulty of diffusion policy optimization is how to obtain the optimal action samples. QVPO finds the optimal action samples via multiple sampling and selection, which can be done parallelly. In contrast, the previous diffusion-based RL method DIPO finds the optimal action sample by performing multiple gradient updates on the actions in the replay buffer. However, the multiple gradient updates cannot be implemented in parallel. In that case, QVPO can speed up the training with the parallelism of GPU. The concrete parallel settings of QVPO in training and in inference are shown in Table 3 and lines 300-301 of the original paper respectively. The action selection number (i.e., the number of parallelly generated actions) is 64 in training and 32 in inference for each state.
Thanks for your comments as well.

**Q7**: I believe the proof is valid. However, could you clarify what $\rho_0$ represents? Is it the discounted state distribution [1]? Additionally, could you explain the reasoning behind assuming the new policy in this manner? Do the authors have any intuition about this assumption?
 
 **A7**: As mentioned in lines 126-129 of our original paper, $\rho_0$ is the distribution of the initial state $s_0$ rather than the discounted state distribution. The discounted state distribution is represented by $d_\pi$. 
 
The assumption is straightforward. We will present a simpler formulation to make it easy to follow up. Generally, we can write the updated policy as $\pi_{new}(a\mid s) = (1-\beta(s))\pi_{old}(a\mid s) + 
\beta(s)\pi(a; a^\star\mid s)$ (similar update formula can be found in equation 4.1 of [R13]), where $\pi(a; a^\star\mid s)$ is a normalized probability distribution which is close to a Dirac distribution on $a^\star \mid s$ (i.e. we use $\frac{I_{a\in \mathcal{N}(a^\star\mid s, \epsilon)}(a)}{S_{\mathcal{N}(a^\star\mid s, \epsilon)}}$ to denote it). Considering weight $A_{\pi_{old}}(s, a^\star)$ in the objective, we just assume the degree of updating is proportional to the weight (i.e., $\beta(s)\propto A_{\pi_{old}}(s, a^\star)$). This assumption is reasonable that policy can achieve larger update steps in states with relatively larger weights.

In that case, we will finally achieve $\pi_{new}(a\mid s) \approx (1-p_{data}(s)A_{\pi_{old}}(s, a^\star)\eta)\pi_{old}(a\mid s) + p_{data}(s) A_{\pi_{old}}(s, a^\star) \eta \pi(a; a^\star\mid s)$.
 
 
**Q8**: I appreciate the explanation regarding reward tuning. One of my main questions following this is why not use $\exp(Q)$ or $\exp(A)$, which is a standard transformation when applying AWR-style methods, as referenced in [2,3]. This approach naturally guarantees that the weight is non-negative.

**A8**: That is a good question. It can be observed that AWR [R3] converges slowly according to Figure 3 of AWR [R3]. One reason for the slow convergence is that $\exp(A)$ is too conservative for policy in online RL. For instance, if there exist 10 different actions for training, we have $A(s,a_1)=1$ and $A(s,a_i)=0.1, i=2,\cdots,10$, the weight of optimal action $a_1$ is $exp(1)\approx 2.7$ and the weight of sub-optimal actions is $\sum_{i=2}^{10}\exp(0.1)\approx 9.9$. In that case, the updated policy with $\exp$ weight function still has a high probability to output a sub-optimal action. In contrast, QVPO avoids this problem via the action selection procedure, which only uses the optimal action to train the diffusion policy. Another problem of $\exp(A)$ is the numerical instability. In practice, if the scale of reward is not properly set, the numerical instability will happen with $\exp(A)$ (e.g., $A(s, a)=10, exp(A(s, a))=22026$). 

[R13] Kakade S, Langford J. Approximately optimal approximate reinforcement learning[C]//Proceedings of the Nineteenth International Conference on Machine Learning. 2002: 267-274.
Thank you for the authors' detailed response. Most of my concerns have been addressed, but a few questions remain unresolved.

1. I believe the proof is valid. However, could you clarify what $\rho_0$ represents? Is it the discounted state distribution [1]? Additionally, could you explain the reasoning behind assuming the new policy in this manner? Do the authors have any intuition about this assumption?

2. I appreciate the explanation regarding reward tuning. One of my main questions following this is why not use $exp(Q)$ or $exp(A)$, which is a standard transformation when applying AWR-style methods, as referenced in [2,3]. This approach naturally guarantees that the weight is non-negative.

[1] Sutton, R. S., & Barto, A. G. (1998). Introduction to reinforcement learning. mit press. Cambridge, MA.

[2] Peng, X. B., Kumar, A., Zhang, G., & Levine, S. (2019). Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177.

[3] Kostrikov, I., Nair, A., & Levine, S. (2021). Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169.
Dear Authors,

Thank you for your rebuttal. You mentioned that "QVPO can efficiently leverage GPU parallel computing, and the multiple-sampling and selecting procedure does not significantly impact time compared to gradient-based optimization like DIPO." Could you clarify the parallel settings used to evaluate the training and inference times for your method? Thank you.
I appreciate the response and the effort made by the authors during the rebuttal. However, I still have a couple of questions. First,  the result of continuous bandit problem looks perfect. Can you elaborate more on how the experiment is actually being done, e.g., how do you sample the initial points? Second, the authors claim that QVPO has a high GPU utilization compared to baselines. However, do you use any parallel environment during training? I believe that parallel environments, either on CPU or GPU, have been a common technique in RL training. If you use hundreds of parallel environments, say 256 (a common number that most devices can support), the GPU utilization will be high for baselines too, and therefore the multiple sampling and selection procedure of QVPO will become a bottleneck, let alone GPU simulators with thousands of parallel environments. Third, the authors claim that QVPO can leverage the multimodality of diffusion policies (line 60), however, the authors didn't prove it or even mention it in the rest of the paper. I feel like this claim is very strong and you should verify it in experiments.
Dear Reviewers and AC, 

Thank you for providing valuable comments. We have tried to address most if not all, concerns raised by the reviewers. 

We observe that, except for Kypi, reviewers have no follow-up comments on our responses. Given the upcoming discussion stage deadline, please let us know if you have any further questions. 

Besides, we would like to express our sincere gratitude to reviewer Kypi for recognizing our contribution and raising their score from 3 to 6. 

Thank you and best regards.

Authors
**Q12: Does one training epoch equal one environment step in your paper?**:

Thanks for your feedback. We confirm that one training epoch equals one environment step in our paper. We do not mention this since it is a common setting in the literature of online RL. However, to make our paper clearer, we will replace "training epoch" with "environment step" in the final version.

**Q13: $t$ in Algorithm 1 looks confusing.**:

Thank you for pointing out this ambiguity. In Algorithm 1, each iteration indicates a training epoch and $T$ means the total number of training epochs (i.e., 1e6 in our experiments on 5 mujoco environments). However, $t$ in line 10 does not mean the training epoch but represents different transitions in the replay buffer. We will use different notations for training epoch (environment step) in the final version. 

**Q14: I really need to see some experiments in more difficult environments for me to believe this paper will have "high impact"**:

We need to emphasize that the HumanoidBench environment [R7] we added is difficult enough to present the "high impact" of QVPO. As mentioned in A4, This environment is based on Unitree H1 humanoid robot with **151 state dimension and 61 action dimension**. As shown in paper [R7], **model-free RL methods, such as PPO and SAC cannot even converge in this challenging environment**. However, as shown in **Figure 3 of our attached pdf file, QVPO (although as a model-free method) even surpasses the performance of advanced model-based RL methods (TD-MPC2, Dreamer-v3)**. Notably, the whole-body control of humanoid is truly very very hard! In that case, existing works on humanoid control such as [R11, R12] tend to separate the RL learning on lower body and upper body of the humanoid. However, QVPO can perform very well with whole-body humanoid control (61 action dimensions include the upper and lower body and two hands). Hence, we believe QVPO will be a revolutionary online RL method for complex continuous control tasks. 

[R7] Sferrazza C, Huang D M, Lin X, et al. Humanoidbench: Simulated humanoid benchmark for whole-body locomotion and manipulation[J]. arXiv preprint arXiv:2403.10506, 2024.

[R11] He T, Luo Z, Xiao W, et al. Learning human-to-humanoid real-time whole-body teleoperation[J]. arXiv preprint arXiv:2403.04436, 2024.

[R12] Cheng X, Ji Y, Chen J, et al. Expressive whole-body control for humanoid robots[J]. arXiv preprint arXiv:2402.16796, 2024.
Thanks for the response. I incorrectly assumed that "iterations" in that paper was equal to the number of environment interactions. It appears you are correct that they are training on more data and your results do mostly match the DIPO results. Can you please confirm that in your paper one epoch equals one environment step? I couldn't find any details specifying this in the paper. I actually think replacing 'epoch' with something along the lines of the number of environment steps would get rid of a lot of confusion. At the very least you should make the relationship between epochs and training data more explicit in the paper.

As I was looking into this I noticed that algorithm 1 is confusing. You loop over $t$ from 1 to $T$, but never specify what $T$ is. Is each iteration through that loop an epoch? Also, you use $t$ twice, as it simultaneously refers to the step in the iteration from 1 to $T$, (lines 1-3) but also refers to the index in the batch for the TD target (line 10).

Regardless of the small issues I've raised in this response I'm feeling far more confident about the paper. Upon confirmation that one epoch means one environment step I'll increase my score to 6. The main reason I don't feel a higher score is justified is that I really need to see some experiments in more difficult environments for me to believe this paper will have "high impact", as is the criteria for a score higher than 6.
Thanks for your valuable suggestions as well. We are glad that our responses help you solve most of your concerns. Here are some further illustrations for your concerns.

**Why results of DSAC-T are so much stronger than ours?**

We want to clarify that the difference in performance between DSAC-T [R8] and QVPO merely **comes from different implementation settings**. As shown in Figure 2 of DSAC-T [R8], the x-axis is not the online interaction steps as in our paper (i.e., training epoch) but the training iteration. As shown in Table 3 of DSAC-T [R8], DSAC-T performs one training iteration with **20 online interaction steps** (**sample batch size in original paper**) for off-policy RL methods. In that case, to fairly compare QVPO with DSAC-T, you need to **multiply all the coordinates on the x-axis by 20**. With that operation, you will obtain results similar to ours on SAC. Actually, it is meaningful to use interaction steps (ours) rather than training iterations (DSAC-T) as the x-axis, since the former can show sample efficiency of different algorithms better in the online RL paradigm.

We want to highlight that **our setting is the same as the most existing works**, including SAC [R8], DIPO [R9], QSM [R10], TD-MPC2 [R1], Dreamer-v3 [R2], etc. **Our results on all other baselines including SAC are also consistent with these works**. Please refer to Figure 1 of SAC [R8], Figure 6 of DIPO [R9], Figure 3 of QSM [R10], Figure 12 of TD-MPC2 [R1], etc.


**Why PPO performs poorly on Humanoid and HalfCheetah?** PPO does perform poorly with limited interactions since it is an on-policy RL method. Many previous works such as DIPO [R7], SAC [R9], Dreamer-v3 [R2] also show similar results to ours. Please refer to Figure 6 in DIPO [R7], Figure 1 in SAC [R9], and Figure 14 in Dreamer-v3 [R9]. 

[R8] Duan J, Wang W, Xiao L, et al. DSAC-T: Distributional soft actor-critic with three refinements[J]. arXiv preprint arXiv:2310.05858, 2023. 

[R9] Haarnoja T, Zhou A, Hartikainen K, et al. Soft actor-critic algorithms and applications[J]. arXiv preprint arXiv:1812.05905, 2018. 

[R10] Psenka M, Escontrela A, Abbeel P, et al. Learning a diffusion model policy from rewards via q-score matching[J]. arXiv preprint arXiv:2312.11752, 2023.
Thanks for your detailed responses. You addressed most of my concerns about the paper, but I was still not 100% convinced by your results. I thought it was odd that PPO does so poorly on Humanoid and HalfCheetah so I looked around for other papers running the same baselines on the same benchmark tasks. I found a paper titled "DSAC-T: Distributional Soft Actor-Critic with Three Refinements" from Duan et al. which also runs several of your baselines on your benchmark tasks. Notably, their implementation of SAC does at least as well as QVPO on all environments except Hopper, and actually does substantially better on Walker, Halfcheetah and Humanoid. This pattern hold across their implementation of several baselines you compare to. This leads me to question whether your method is truly stronger than these baselines or if the difference in performance that you observe is merely a matter of implementation details. In order to improve my score it would be helpful to understand why their results are so much stronger than yours.