Diffusion Actor-Critic with Entropy Regulator
Authors: Yinuo Wang
Likun Wang
Yuxuan Jiang
Wenjun Zou
Tong Liu
Xujie Song
Wenxuan Wang
Liming Xiao
Jiang WU
Jingliang Duan
Shengbo Eben Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion model
online reinforcement learning
Gaussian mixture model
Primary area: reinforcement_learning

Abstract: Reinforcement learning (RL) has proven highly effective in addressing complex decision-making and control tasks. However, in most traditional RL algorithms, the policy is typically parameterized as a diagonal Gaussian distribution with learned mean and variance, which constrains their capability to acquire complex policies. In response to this problem, we propose an online RL algorithm termed diffusion actor-critic with entropy regulator (DACER). This algorithm conceptualizes the reverse process of the diffusion model as a novel policy function and leverages the capability of the diffusion model to fit multimodal distributions, thereby enhancing the representational capacity of the policy. Since the distribution of the diffusion policy lacks an analytical expression, its entropy cannot be determined analytically. To mitigate this, we propose a method to estimate the entropy of the diffusion policy utilizing Gaussian mixture model. Building on the estimated entropy, we can learn a parameter $\alpha$ that modulates the degree of exploration and exploitation. Parameter $\alpha$ will be employed to adaptively regulate the variance of the added noise, which is applied to the action output by the diffusion model. Experimental trials on MuJoCo benchmarks and a multimodal task demonstrate that the DACER algorithm achieves state-of-the-art (SOTA) performance in most MuJoCo control tasks while exhibiting a stronger representational capacity of the diffusion policy.

Diffusion Actor-Critic with Entropy Regulator
Yinuo Wang'
Likun Wang'
Yuxuan Jiang'
Wenjun Zou'
Tong Liu'
Xujie Song'
Wenxuan Wang'
Liming Xiao² Jiang Wu?
Jingliang Duan 1.2+
Shengbo Eben Li
School of Vehicle and Mobility, Tsinghua University School of Mechanical Engincering, University of Science and Technology Beijing
Abstract
Reinforcement learning (RL) has proven highly effective in addressing complex decision-makee and control tasks. However, most traditional RL algorithms, the policy is typically parameterized as a diagonal Gaussian distribution with learned mean and variance, which constrains their capability to acquire complex policies. In response to this problem, we propose an online RI algorithm termed diffusion actor-critic with entropy regulator (DACER). This algorithm conceptualizes the novel policy function and leverages the
reverse process of the diffusion model capability the diffusion model to fit multimodal distributions, thereby enhancing the representational capacity of the policy. Since the distribution of the diffusion policy lacks an analytical expression, its entropy cannot be determined analytically. To mitigate this, propose method to estimate the entropy of the diffusion policy utilizing Gaussian mixture model. Building on the estimated entropy, we learn parameter that modulates the degree of exploration and exploitation. Parameter a will be employed to adaptively regulate the variance of the added noise, which is applied to the action output by the diffusion model. Experimental trials on MuJoCo benchmarks and a multimodal task demonstrate that the DACER algorithm achieves state-of-the-ar (SOTA) performance most MuJoCo control tasks while exhibiting stronger representational capacity of the diffusion policy.
Introduction
Recently, deep reinforcement learning (RL) has emerged an effective method for solving optimal control problems in the physical world [14, 29, 22, 25]. In most existing RL algorithms, the policy is parameterized as deterministic function or a diagonal Gaussian distribution with the learned mean and variance [31, 32, 16, 10]. However, the theoretically optimal policy may exhibit strong multimodality, which cannot be well modeled by deterministic or diagonal Gaussian policies [43, 21,44 45]. Restricted policy representation capabilities can make algorithms prone to local optimal solutions, damaging policy performance. For instance, in situations where two distinct actions in the same state yield approximately the same Q-value, the Gaussian policy approximates the bimodal action by maximizing the Q-value. This results in the policy displaying mode-covering behavior, concentrating high density in the intermediate region between the two patterns, which inherently low-density region with lower Q-value. Consequently, modeling the policy with unimodal Gaussian distribution is likely to significantly impair policy learning.
Lately, the diffusion model has become widely known as a generative model for its powerful ability to fit multimodal distributions [18, 35, 8]. It learns the original data distribution through the idea of stepwise addition and removal of noise and has excellent performance in the fields of image [46, 28] and video generation [9, 3]. The policy network in RL can be seen as a state-conditional generative
Cooresponding author <duanjl15@ 163.com> <lishbo@ ctiabbo@tsingguu.cdu......
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

model. Given the ability of diffusion models to fit complex distributions, there is increasing work on combining RL with diffusion models. Online RL learns policies by interacting with the environment (16,32) Offline RI, also known as batch RI., aims to effectively learn policies from previously collected data without interacting with the environment [1, 4]. In practical applications, many control problems have excellent simulators. At this time, using offline RL is not appropriate, as online RL with interaction capabilities performs better. Therefore, this paper focuses on how the diffusion model can be combined with online RL.
In this work, we propose diffusion actor-critic with entropy regulator (DACER), a generalized new approach to combine diffusion policy with online RL. Specifically, we base DACER on the denoising diffusion probabilistic model (DDPM) [18]. recent work by He et al. [44] points out that the representational power of diffusion models stems mainly from the reverse diffusion processes, not from the forward diffusion processes. Inspired by this work, we reconceptualize the reverse process of the diffusion model as a novel policy approximator, leveraging its powerful representation capabilitiess to enhance the performance of RL. RL. algorithms. The optimization objective of this novel policy function is to maximize the expected Q-value. Maximizing entropy is important for policy exploration in RL, but the entropy of the diffusion policy is difficult to determine. Therefore, we choose to sample actions at fixed intervals and use a Gaussian mixture model (GMM) to fit the action distributions. Subsequently. We can calculate the approximate entropy of the policy in each state. The average of these entropies is then used as approximation of the current diffusion policy entropy. Then, use the estimated entropy to regulate the degree of exploration and exploitation of diffusion policy.
In summary, the key contributions of this paper are the following: 1) We propose to consider the reverse process of the diffusion model as a novel policy function. The objective function of the diffusion policy is to maximize the expected Q-value and thus achieve policy improvement. 2) We propose a method for estimating the entropy of diffusion policy. The estimated value is utilized to achieve an adaptive adjustment of the exploration level of the diffusion policy, thus improving the policy performance. 3) We evaluate the efficiency and generality of our method on the popular MuJoCo benchmarking. Compared with DDPG [33]. TD3 [12], PPO [32], SAC [16]. DSAC [11, 10], and TRPO [31]. our approach achieves the SOTA performance. In addition, we demonstrate the superior representational capacity of our algorithm through specific multi-goal task. 4) We provide the DACER code written in JAX to facilitate future researchers to follow our work!
Section 2 introduces and summarizes existing approaches to diffusion policy in offline RL and online RL, pointing out some of their problems. Section 3 provides an introduction to online RL and diffusion models. Our approach to combining diffusion policy with the mainstream actor-critic framework, as well as methods to enhance the performance of diffusion policy will be presented in section 4. The results of the experiments in the MuJoCo environment, the ablation experiments as well as the multimodality task will be presented section 5. Section 6 provides the conclusions of this paper.
2 Related Work
Diffusion Policy in Offline RL Offline RL leverages pre-collected datasets for policy development, circumventing direct environmental interaction. Current offline RL research utilizing diffusion models as policy networks primarily adhere to the behavioral cloning framework [7, 26]. Within this framework, two main objectives emerge: performance enhancement and training efficiency improvement. For the former, Cheng et al. [6] proposed a Diffusion Policy, casting the policy as a conditional denoising diffusion process within the action space to accommodate complex multimodal action distributions. Wang el al. [43] introduced Diffusion-QIL, which integrates behavior cloning via diffusion model loss with Q-learning for policy improvement. Ajay al. [2] created Decision Diffusion, incorporating classifier-free guidance into the diffusion model to integrate trajectory information, such as rewards and constraints. Addressing the latter, Kang et al. [20] developed efficient diffusion policies (EDP), an evolution of Diffusion-QL. EDP accelerates training by utilizing initial actions from state-action pairs in the buffer and applying a one-step sample for final action derivation. Chen et al. [5] proposed a consistency policy that enhances diffusion algorithm efficiency through one-step action generation from noise during training and inference. Although Diffusion Policy's powerful ability to fit multimodal policy distributions can achieve good performance in
hhunss/giihibbcommhpppss/aDDDDD DiAEEER-DDiuii---iiiii Oith--Onlin-RU

---

offline RL tasks, this method of policy improvement based on behavioral cloning cannot be directly transferred to online RL. In addition, the biggest challenge facing offline RL, the distribution shift problem, has not been completely solved. This paper focuses on online RI,, moving away from the framework of behavioral cloning.
Diffusion Policy in Online RL Online RL, characterized by real-time environment interaction, contrasts with offline RL's dependence on pre-existing datasets. To date, only two studies have delved into integrating online RL with diffusion models. Yang et al. [45] pioneered this approach by proposing action gradient method. This approach achieves policy improvement by updating the action in the replay buffer through the VaQ, followed by mimicry learning of the action post-update using a diffusion model. However, action gradient increased the additional training time. Furthermore, it is difficult to fully learn both the action gradient and imitation learning steps simultaneously. which also resulted in suboptimal performance of this method in MuJoCo tasks. Psenka et al. [30] proposed Q-score matching (QSM), a new methodology for off-policy reinforcement learning that leverages the score-based based structure of diffusion model policies align with the V. This approach aims to overcome the limitations of simple behavior cloning in actor-critic settings by integrating the policy's score with the Q-function's 's action gradient. However, QSM needs accurately learn Q in most of the action space to achieve optimal guidance. This is difficult to accomplish, resulting in suboptimal performance of QSM.
Our method is motivated to propose a diffusion policy that can be combined with most existing actor-critic frameworks. We first consider the reverse diffusion process of the diffusion model policy function with strong representational power. Then, we use the entropy estimation method to balance the exploration and utilization of diffusion policy and improve the performance of the policy.
Comparison with Diffusion-QL. Diffusion-QU [43] made a successful attempt by replacing the diagonal Gaussian policy with a diffusion model. It also guides the updating of the policy by adding the normalized Q-value in the policy loss term. The main differences between our work and Diffusion- QL are as follows: 1) Diffusion- -L is still essentially an architecture for imitation learning, and policy updates are mainly motivated by the imitation learning loss term. 2) Our work adaptively regulates the standard deviation of random noise in the sampling process a = a a+ - (0, I), where a learned parameter, exploitation and subsequently enhances the performance of the diffusion policy. Ablation experiments provide evidence supporting these findings.
a hyperparameter. This method effectively balances exploration and
3 Preliminaries
3.1 Online Reinforcement Learning
In the conventional framework of RL, interactions between the agent and its environment occur in sequential discrete time steps. Typically, the environment is modeled Markov decision process (MDP) with continuous states and actions [38]. The environment provides feedback through a bounded reward function denoted by r(st, az). The likelihood of transitioning new state based on the agent's 's action is expressed by the probability p(*++1|5>, 0z). State-action pairs for the current and next steps indicated as (s, a) and (s/, a'). The decision-making of agent at any state 8, is guided by a stochastic policy π (ay st), which determines the probability distribution over feasible actions that state.
In the realm of online RL, agents engage in real-time learning and decision meciiion-mmakin through direct interactions with their environments. Such interactions are captured within tuple (St, ar, Pt, 8t+1), representing the transition during each interaction. It is common practice to store these transitions in experience replay buffer, symbolized as B. Throughout the training phase, random samples drawn from B produce batches of data that contribute to a more consistent training process. The fundamental aim of traditional online RL. strategies is craft policy that optimizes the expected total reward:
20
J= = E Essgeragee)
Σ
i x (s,,, at)
(1)
3

---

where y E (0, represents the discount factor. The Q-value for a state- actton pair (s, a) is given by
00
Q(s,a)=BB E
Σ ==
r'r(s,, a;) = s, 0, =
(2)
RL typically employs actor-critic framework [25, 24], which includes both policy function, symbolized by я, and a corresponding Q-value function, noted as Q". The process of policy iteration often used to achieve the optimal policy æ², cycling through phases of policy evaluation and enhancement. In the policy evaluation phase, the Q-value Q is recalibrated according to the self-consistency requirements dictated by the Bellman equation:
Q" r(s,a) = (2"(s',
In the policy improvement phase, an enhanced policy nnw is sought by optimizing current Q-value Quuld:
Tnew arg max Sende,ann 1Q (s, a)].
In practical applications, neural networks are often used parameterize both the policy and value functions, represented by and Qs. respectively. These functions are refined through the application of gradient descent methods aimed at reducing the loss functions for both
critic, (0) Be (r(3,aa) + (0 a')) (5) and the actor,
(Ø) -Band, a~r (Qaald (8, a)). These loss functions are structured based on the principles outlined in (3) and (4).
3.2 Diffusion Models
Diffusion models [34, 18, 36, 37] are highly effective generative tools. They convert data from its original distribution to a Gaussian noise distribution by gradually adding noise and then reconstruct the data by gradually removing this noise through reverse process. This process is typically described as continuous Markov chain: the forward process incrementally increases the noise level, while the reverse process involves conditional generative model trained to predict the optimal reverse transitions at each denoising step. Consequently, the model reverses the diffusion sequence to generate data samples starting from pure noise.
Let us define pa (2r)) := f pa (Zo.T dæ where Z1...., TT denote latent variables sharing the same dimensionality as the data variable xn ~ g(zo), where g(xo) means original data distribution. In forward diffusion chain, the noise incrementally introduced to the data æo ((x0) across T steps, adhering to predetermined variance sequence denoted by Be, described as
(
II (m))) (( N( v BL 3.)
(5)
When T + oo, xT distributes as an isotropic Gaussian distribution [21]. The reverse diffusion process of the diffusion model be represented
po(xoTT) = p((r)) II (p:((=112)))) po(a 12x.)
1), Eo ()),
(6)
where p(æT) Nx(T;; 0, I) under the condition that IT= - -Bc) 0.
4
Method
In this section, we detail the design of our diffusion actor-critic with entropy regulator (DACER). First, we consider the reverse diffusion process of the diffusion model as a new policy approximator, serving as the policy function in RL. Second, We directly optimize the diffusion policy using gradient descent, whose objective function is to maximize expected Q-values. This feature allows it to be integrated with mainstream RL algorithms that do not require entropy. However, the diffusion policy

---

learned this way produces overly deterministic actions with poor performance. When attempting integrate the maximization entropy RL framework, we find the entropy the diffusion policy is difficult to analytically determine. Therefore, we use GMM to approximate the entropy of the diffusion policy, and then learn a parameter a based on it to adjust the exploration level of diffusion policy.
4.1 Diffusion Policy Representation
We the reverse process of conditional diffusion model as a parametric policy:
T
To( 8) (u(orrlss) = p((r))))))))) II (0:-11aa,.00 8):
where p(ar) = [N,, I), the end sample the reverse chain, ag. is the action used for RL evaluation. Generally, ps p=(a,-1la;s)) could be modeled (aa=-1; 8, t) Eg Ea,a, 8, t)). We choose to parameterize To (a(a)s) like DDPM [18], which sets Ea s) I I to fixed time-dependent constants, and constructs the mean μ4 from noise prediction model as
Gaussian distribution
1 Var
B. VI
Mo (o 8,t)
a.
co co 8,t)
(8)
ai
where 0z ār = IT Okk, and Eg a parametric model.
To obtain an action from DDPM, we need to draw samples from T different Gaussian distributions sequentially. The sampling process be reformulated
1 Vat
3e VI
en (ara,, 8.1) + VBr,
a-
with the reparametrization trick, where NO0,, I), is the reverse timestep from T ar NO I).
4.2 Diffusion Policy Learning
In integrating diffusion policy with offline RI., policy improvement relies minimizing the behavior- cloning term. However, in online RI., without a dataset to imitate, we discarded the hehavior-cloning term and the imitation learning framework. In this study, the policy-learning objective is to maximize the expected Q-values of the actions generated by the diffusion network given the state:
max Eg ag (s)))
(10)
Unlike the traditional reverse diffusion process, our study requires recording the gradient of the whole process. The gradient of the Q-value function with respect the action is backpropagated through the entire diffusion chain.
Policy improvement is introduced above; next, we introduce policy evaluation. The Q-value function learned through a conventional approach, which involves minimizing the Bellman operator [13, 25, 38] with the double Q-learning trick [40]. We built two Q-networks Qø, (s, a), Qo (s, a). and target network Qa; (s, a), Qag (s, Then we give the objective function of policy evaluation, which shown as
((r(s.a)+ + min (s.(s",a') 2 -Qu(s,,))
min ((
(11)
1=1.2
where a' is obtained by inputting the s' into the diffusion policy, B means replay buffer. Building on this, we employ the tricks in DSAC [11, 10] to mitigate the problem of Q-value overestimation.
The diffusion policy we construct can be directly combined with mainstream RL algorithms that do not require policy entropy. However, training with the above diffusion policy learning method suffers from overly deterministic policy actions, resulting in poor performance of the final diffusion policy, In the next section, we will propose entropy estimation to solve this problem and obtain diffusion policy with SOTA performance.
5

---

4.3 Diffusion Policy with Entropy
The diffuuion policy's 's distribution lacks an analytic expression, we cannot directly determine its entropy. However, in the same state, we can use multiple samples to obtain a series of actions. By fitting these action points, we can estimate the action distribution corresponding the state.
In this paper, we use Gaussian mixture model (GMM) to fit the policy distribution. The GMM forms complex probability density function by combining multiple Gaussian distributions, which can be represented as
K
f(a) = Σ wk N(alpu Σx),
(12)
k=1
where K is the number of Gaussian distributions, and wk is the mixing weight of the k-th component, satisfying Σ wk 1, wi 0. Mg. En are the mean and covariance matrices of the k-th Gaussian distribution, respectively.
For each state, Maximization algorithm is then used to estimate the parameters of the GMM. In the expectation step, the posterior probability that each data point a' belongs each component & is computed, denoted as
diffusion policy to sample N actions, a' a',a² a ..... aN E A. The Expectation-
Wk Maaa) Mg. Ex) (=%) K Σ wy Naall E,)
(13)
where 7(2)) denotes that under the current parameter estimates, the observed data a' come from the k-th component of the probability. In the maximization step, the results of the Eq. (13) calculations are used to update the parameters and mixing weights for each component:
1 N
N
(21)).m = >E)) a' E E (2
-4 -4))
1T)TT
W7k
(14)
E
Iterative optimization continues until parameter convergence. Based on our experimental experience in the MuloCo environments, a general setting of 3 provides a better to the action distribution.
According to Eq. (12), we can estimate the entropy of the action distribution corresponding to the state by [19]
H
Wk log wk +
Σ wk log ((2me)* Exl) R=1
(15)
where d the dimension of action. Then, the mean the entropy of the actions associated with the chosen batch of states used as the estimated entropy H. of the diffusion policy.
Similar to maximizing entropy RL, we learn parameter based on the estimated entropy. We
update this parameter using
a a Ba[ H.,
(16)
where H is target entropy. Finally, we during training, where is a hyperparameter and a is the output of diffusion policy. Additionally, no noise is added during the evaluation phase. We summarize implementation in Algorithm 1.
a Xa AO I) adjust the diffusion policy entropy
5 Experiments
We evaluate the performance of our method some control tasks of RL within MuloCo [39]. The benchmark tasks utilized in this study are depicted in Fig. 5, including Humanoid-v3, Ant-v3, Hopper-v3, Pusher-v2, and Swimmer- v3. Moreover, we conducted experiments in multi-goal task to demonstrate the excellent representa- tional and exploratory capabilities of our diffusion policy. We also provide ablation studies on the critical components for better understanding. All baseline algorithms available in GOPS [42], an open-source RL solver developed with PyTorch
HalfCheetah-v3, Walker2d-v3,
Baselines. Our algorithm is compared and evaluated against the six well-known model-free algo- rithms. These include DDPG [33], TD3 [12], PPO [32], SAC [16], DSAC (11, 10], and TRPO [31]. These basclines have been extensively tested and applied in series of demanding domains.

---

Algorithm Diffusion Actor-Critic with Entropy Regulator for Online RL
Input: X, 0, Ø1, Ø2, ø1, 62 a, Bqs Bas Bg, and for each iteration do
for each sampling step do
Sample To( $) by Eq. (7)
Add noise a = a + Xa (O,
Get reward r and new state s'
Store batch of samples (8, a, r, 8') in replay buffer B end for
for cach update step do
Sample data from B Update critic networks using o; + ó; - -
(o.) for [1,2)
Update diffusion policy network using 0 0 B Vo (0) if step mod 10000 == 0 then
Estimate the entropy of diffusion policy H E [7x]
Update using Eq. (16)
Update target networks using 0 = paj (1 p)gs for i = {1,
end for end for
Experimental details. To ensure fair comparison, we incorporated the diffusion policy as policy approximation function within GOPS and implemented DACER with JAX, which improves training speed by 4-5 times compared Py ToTocch while maintaining consistent performance. All algorithms and tasks use the same three-layer MLP neural network with GeLU [17] or Mish [27] activation functions, the latter used only for the noise prediction network in the diffusion policy. Initially, we encode timestep into 16 dimensions using sinusoidal embedding [41], then merge this encoded result with the state and action a, during the current denoising step, and input into the prediction noise network to generate the output. The impact of the reverse diffusion step size, T, on the experimental results will be examined in the ablation experiments. T is set to 20 for all experiments eventually. The Adam [23] optimization method is employed for all parameter updates. In this paper, the total training step size for all experiments is set at 1.5 million, with the results of all experiments averaged over five random seeds. The CPU used for the experiment is the AMD Ryzen Threadripper 3960X 24-Core Processor, and the GPU is NVIDIA GeForce RTX 3090Ti. Taking Humanoid-v3 example, the time taken to train 1.5 million in the JAX framework is 6 hours. More detailed hyperparameters are provided in Appendix A.2 due to space limits.
Evaluation Protocol. In this paper, we use the same assessment metries as DSAC. For each seed, the metric is derived by averaging the highest return values observed during the final 10% of iteration steps in each run, with evaluations conducted every 15, 15,000 iterations. Each assessment result the average of ten episodes. The results from the five seeds are then aggregated to calculate the mean and standard deviation. Additionally, the training curves in Fig. provide insights into the stability the training process.
5.1 Comparative Evaluation
Each algorithm was subjected to five distinct tests, utilizing a variety consistent random seeds to ensure robustness in the results. Fig. and Table 1 display the learning curves and performance strategies, respectively. Our comprehensive findings reveal that across all evaluated tasks, the DACER algorithm consistently matched or surpassed the performance of all competing benchmark algorithms. Specifically, in the Humanoid-v3 scenario, our algorithm demonstrated enhancements of 124.7%. 111.1%, 73.1%, 27.3%, 9.8%, and 1131.9% over DDPG, TD3, PPO, SAC. DSAC, and TRPO, respectively.
5.2 Policy Representation Experiment
In this section, we conduct an experiment to confirm the representation capability of the diffusion policy. We use an environment called "Multi-goal" [15], shown in Fig, 2, where the z-axis and
7

---

(a) Humanoid-v3
(b) Ant-v3
HalfCheetah-v2
(d) Walker2d-v3
(c) Inverted2Pendulum- v3
(f) H3pper-v3
(g) Pusher-v2
(h) Swimmer-v3
Training curves on benchmarks. The solid lines represent the mean, while the shaded regions indicate the 95% confidence interval over five runs. The iteration of PPO and TRPO is measured by the number of network updates.
Figure 1:
Table 1
Average final return. Computed as the mean of the highest return values observed in the final 10% of iteration steps per run, with evaluation interval of 15,000 iterations. The maximum value for each task is bolded. corresponds to standard deviation over five runs.
Task Humanoid-v3
DACER
DSAC
DDPG
TRPO
PPO
11888 : 244
10829 243
9335 :: 695
5631 = 435
5291 1 662
965 :: 555
6869 =: 1563
Ant-v3
9108 103
7086 i 261
6427 1 804
6184 486
4549 788
6203 578
61.56 185
Halfcheetah-v3
17177
8632 4041
13970 2083
967
5789
Walker2d-v3
6701 : 62
6424 + 147
6200 :: 263
$237 + 335
4095 :: 68
5502 591
4831 637
Invensddoubleeencaluus
9360 +
9360 0
+ 0
9347 15
9183 =
6259 2065
9356 +
Hopper-y3
4104 49
3660+5333
2483 + 943
3569 + 455
2644 659
3474 400
2647 482
Pusherrv2
-19 + 1
-19
-20
-21 #1
-30 = 6
-23
-23
Swimmer-y3
70
130
y-axis represent 2D states. In this setup, the agent is represented as a 2D point mass situated on a 7 plane. The objective for the agent is to navigate towards one of four symmetrically positioned points: (0, 5). (0, 5). (5, 0), and (-5,0) 0). Since the goal positions are symmetrically distributed at the four points, policy with strong representational capacity should enable the Q-function to learn the four symmetric peaks across the entire state space. This result reflects the policy's 's capacity for exploration in understanding the environment.
We compare the performance of DACER with DSAC, TD3, and PPO, as shown in Fig. 2. The results show that DACER's actions likely point to the nearest peak in different states. DACER's value function curve shows four symmetrical peaks, aligning with the previous analysis. Compared to DSAC, our method learns better policy representation, mainly due using diffusion policy instead of an MIP. In contrast, TD3 and PPO generate more random actions with poorer policy representation, lacking the symmetrical peaks in their value function curves. Overall, our method demonstrates superior representational capability.
To demonstrate the powerful multimodality of DACER, select five points requiring multimodal policies: (0.5, 0.5), (0.5, -0.5), (-0.5, -0.5), (-0.5, 0.5), and (0, 0). For each point, we sampled 100 trajectories. The trajectories
plotted in Fig. 3. The results show that compared with DSAC, DACER exhibits strong multimodality. This also explains why only the Q-function of DACER can learn the nearly perfectly symmetrical four peaks.
5.3 Ablation Study
In this section, we analyze why DACER outperforms other baseline algorithms on MuloCo tasks. We conduct ablation experiments to investigate the impact of the following three aspects the
8

---

(a) DACER
(b) DSAC
(c) TD3
(d) PPO
Figure 2: Policy representation comparison different policies on a multimodal environment. The first row exhibits the policy distribution. The length of the red arrowheads denotes the size of the action vector, and the direction of the red arrowheads denotes the direction of actions. The second row shows the value function of each state point.
Figure Multi-goal multimodal experiments. We selected points that require multimodal policies: (0, 0), (-0.5, 0.5), (0.5, 0.5), (0.5, -0.5), (-0.5, -0.5), and sampled 100 trajectories for each point. The top row shows the experimental results of DACER, another shows the experimental results of DSAC.
performance of the diffusion policy: 1) whether adding Gaussian noise the final output action of the diffusion policy; 2) whether the standard deviation of the added Gaussian noise can be adaptively adjusted by estimated entropy; 3) different reverse diffusion step size
Only Q-learning. In section 4.2, we propose a method using the reverse diffusion process as policy approximator, which can be combined with the non-maximizing entropy RL algorithm. However, the diffusion policy trained without entropy exhibits poor exploratory properties, leading to suboptimal performance. Using Walker2d-v3 as example, we compared the training curves of this method with the DACER algorithm, as shown in Fig. 4(a).
Fixed and linear decay noise factor. In order to verify that using the estimated entropy to adaptively adjust the noise factor plays an important role the final performance, we conducted the following two experiments in the Walker2d-v2 task: 1) Fixed noise factor to 0.1: 2) The noise factor starts

---

from 0.27 and linearly decreases to 0.1 during the training process. These two values were chosen because the starting and ending noise factor for adaptive tuning in this setting is about in this range. As shown in Fig. 4(b), our method of adaptively adjusting the noise factor based on the estimated entropy achieves the best performance.
Diffusion steps. We further examined the performance of the diffusion policy the number of diffusion timesteps T varied. We used the Walker2d-v3 task to plot training curves for T 10, 20, and 30, as shown in Fig. 4(c). Experimental results indicate that a larger number of diffusion steps does not necessarily lead to better performance. Excessive diffusion steps can cause gradient explosion, significantly reducing the performance of diffusion policy. After balancing performance and computational efficiency, we selected 20 diffusion steps for all experiments.
7000
2000
800
8000
2010
1000
(a) Ablation for the entropy regulator mechanism.
(b) Ablation for the noise factor
(c) Ablation for the different
modulation mechanism.
diffusion steps.
Figure 4: Ablation experiment curves. (a) DAC stands for not using the entropy regulator. DACER's performance Walker2d-v3 far better than DAC. (b) Adaptive tuning of the noise factor based the estimated entropy achieved the best performance compared to fixing the noise factor or using the adaptive tuning method with initial, end values followed by linear decay method. (c) The best performance was achieved with diffusion steps equal to 20, in addition to the instability of the training process when equal 30.
Conclusion
In this study, we propose the diffusion actor-critic with entropy regulator (DACER) algorithm, novel RL method designed to overcome the limitations of traditional RI. methods that use diagonal Gaussian distributions for policy parameterization. By utilizing the inverse process of the diffusion model, DACER effectively handles multimodal distributions, enabling the creation of more complex policies and improving policy performance. A significant challenge arises from the lack of analytical expressions to determine the entropy of diffusion strategy. To address this, employ GMM to estimate entropy, thereby facilitating the learning of a key parameter, a, which adjusts the exploration- exploitation balance by regulating the noise variance in the action output. Empirical tests the MuJoCo benchmark and a multimodal task show the superior performance of DACER.
7
Acknowledgements
This study is supported by National Key R&D Program of China with 2022YFB2502901 and Tsinghua University Initiative Scientific Research Program.
10

---

References
[1] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference Machine Learning, pages 104-114, PMLR, 2020.
[2] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision-making? The Fleventh International Conference on Learning Representations, 2023.
[3] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311. 1552222711.. 2023.
[4] David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline without off-policy evaluation. Advances in Neural Information Processing Systems, 34:4933-4995
2021.
[5] Yuhui Chen, Haoran Li, and Dongbin Zhao. Boosting continuous control with consistency policy. arXiv preprint arXiv:2310.06344 2023.
[6] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arX1v:2303.04137, 2023.
[7] Felipe Codevilla, Eder Santana, Antonio M López, and Adrien Gaidon. Exploring the limitations of behavior cloning for autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9329-93333 2019.
[8] Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. Diffusion models in vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[9] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
[10] Jingliang Duan, Yang Guan, Shengbo Eben Li, Yangang Ren, Qi Sun, and Bo Cheng. Distribu- tional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors. IEEE Transactions on Neural Networks and Learning Systems, (3(11):6584-658 2021.
[11] Jingliang Duan, Wenxuan Wang. Liming Xiao, Jiaxin Gao, and Shengbo Eben Li. Dsac-t: Distributional soft actor-critic with three refinements. arXiv preprint arXiv:2310.05858, 2023.
[12] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, pages 1587-1596. PMLR, 2018
[13] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference Machine Learning, pages 2052-2062. PMLR, 2019.
[14] Yang Guan, Yangang Ren, Qi Sun, Shengbo Eben Li, Haitong Ma, Jingliang Duan, Yifan Dai, and Bo Cheng. Integrated decision and control: Toward interpretable and computationally efficient driving intelligence. IEEE Transactions on Cybernetics, 53(2):859- 873, 2022.
[15] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning, pages 1362-1361. PMLR, 2017.
[16] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pages 1861-1870. PMLR, 2018.
[17] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415,000 2016.

---

[18] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 186640068551 2020.
[19] Marco F Huber, Tim Bailey, Hugh Durrant- Durrant-Whyte, and Uwe D Hanebeck. On entropy ap- proximation for gaussian mixture random vectors. In 2008 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, pages 181-188. IEEE, 2008
[20] Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. Conference and Workshop on Neural Information Processing Systems, 2023.
[21] Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[22] Elia Kaufmann, Leonard Bauersfeld, Antonio Loquereio, Matthias Müller, Vladlen Koltun, and Davide Scaramuzza. Champion-level drone racing using deep reinforcement learning. Nature, 620(7976):882-987 2023.
[23] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[24] Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in Neural Information Processing Systems, 12, 1999.
[25] Eben Li. Reinforcement Learning for Sequential Decision and Optimal Control. Springer Verlag, Singapore, 2023.
[26] Abdoulaye O Ly and Moulay Akhloufi. Learning to drive by imitation: An overview of deep behavior cloning methods. IEEE Transactions on Intelligent Vehicles, 6(2): 195-2995, 2020.
[27] Diganta Misra. Mish: self regularized non-monotonic activation function. arXiv preprint arXiv: 08681, 2019.
[28) William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195-4205, 2023.
[29] Baiyu Peng, Qi Sun, Shengbo Eben Li, Dongsuk Kum, Yuming Yin, Junqing Wei, and Tianyu Gu. End-to-end autonomous driving through dueling double deep q-network. Automotive Innovation, 4:328-337, 2021.
[30] Michael Psenka, Alejandro Escontrela, Pieter Abbeel, and Yi Ma. Learning diffusion model policy from rewards via q-score matching. arXiv preprint arXi::2312.11752. 2023.
[31] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pages 1889-1897. PMLR, 2015.
[32] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1eeei10770600 06347, 2017.
[33] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International Conference on Machine Learning, pages 387-395. PMLR, 2014.
[34] Jascha Sohl-Dickster Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256-2265. PMLR, 2015.
[35] Yang Song, Conor Durkan, Iain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models, Advances in Neural Information Processing Systems, 34:1415- 1428, 2021.
[36] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution, Advances in Neural Information Processing Systems, 32, 2019.
12

---

[37] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprini arXiv:201 1. 13456, 2020.
[38] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press. 2018.
[39] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: physics engine for model-based control. In Intelligent Robots and Systems, 2012.
[40] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
[41] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances Neural Information Processing Systems, 30, 2017.
[42] Wenxuan Wang, Yuhang Zhang, Jiaxin Gao, Yuxuan Jiang, Yujie Yang, Zhilong Zheng, Wenjun Zou, Jie Li, Congsheng Zhang, Wenhan Cao, et al. Gops: A general optimal control prob- lem solver for autonomous driving and industrial control applications. Communications in Transportation Research, 3:100096, 2023.
[43] Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. The Eleventh International Conference on Learning Representations, 2023.
[44] Saining Xie Xinlei Chen, Zhuang Liu and Kaiming He. Deconstructing denoising diffusion models for self superviedd learning. arXiv preprint arXiv:2401.14444, 2024.
[45] Long Yang, Zhixiong Huang, Fenghao Lei, Yucun Zhong, Yiming Yang, Cong Fang, Shiting Wen, Binbin Zhou, and Zhouchen Lin. Policy representation via diffusion probability model for reinforcement learning. arXiv preprint arXiv:2305. 13122, 2023.
[46] Ruihan Yang, Prakhar Srivastava, and Stephan Mandt. Diffusion probabilistic modeling for video generation. Entropy, 25(10):1469, 2023.
13

---

A Environmental Details
A.1 Experimental Environment Introduction
The benchmark tasks utilized in this study are depicted in Fig. 5, including Humanoid-v3, Ant-v3,
HallCheetah--3,, Walker2d-v3, InvercelDoubbeeennuuuum- Hopper-v3, Pusher-v2, and Swimmer- v3.
(a)
(b)
(c)
(d)
(e)
(h)
Figure 5: Simulation tasks. (a) Humanoid-v3:(a a) E R76 x RDT (b) Ant-v3: (s a)
Rii1, x RR (c) HalfCheetah-v33: (s a) IR17 TR. (d) Walker2d-v3: (s x a) E R277 RR (e)
(s x InveredDaatlePenduuu--3
E R6 TR (f) H3pper-v3: (sx E R x R (g) Pusher-v2:
g23 x RT. (h) Swimmer-v3: (s x a)
E 12 R
A.2 Training Details on MuJoCo tasks
Mujoco [39] is a simulation engine primarily designed for research in RL and robotics. It provides a versatile, physics-based platform for developing and testing various RL algorithms. Core features of Mujoco include a highly efficient physics engine, realistic modeling of dynamic systems, and support for complex articulated robots. Currently, it is of the most recognized benchmark environments for RL and continuous control.
The hyperparameters of all baseline algorithms shown in Table 2. Moreover, the hyperparameters of the DACER in the MuloCo task are shown in Table 3.
Limitation and Future Work
In this study, we propose using GMM to estimate the entropy of the diffusion policy and, based this estimate, learn parameter a to balance exploration and exploitation. However, the process of estimating entropy requires large number of samples and takes long time (about 40 ms). Therefore, we estimate the entropy of diffusion policy every 10,000 iterations to reduce the impact on training time. But, this approach prevents perfect integration with maximizing entropy RL. In future work, we will avoid using batch-sizz size data to estimate entropy and find a balance between estimation accuracy and computational efficiency so to better combine our method with maximizing entropy RL.
C Positive and Negative Social Impact
In this paper, propose DACER, online RI. algorithm that uses the reverse diffusion process as a policy approximator. Diffusion policy has powerful multimodal representation capabilities, making it widely applicable in complex environments such as automated manufacturing, autonomous
14

---

TABLE 2
DETAILED HYPERPARAMETERS.
Hyperparameters
Value
Shared
Replay buffer capacity
1000000
Buffer warm-up size
30000
Batch size
256
Initial alpha a
0.272
Action bound
[-1.1]
Hidden layers in critic network Hidden layers in actor network Activation in critic network
[256, 256, 256]
[256, 256, 256]
GeLU
Activation in actor network
GeLU
Optimizer
Adam (31 = 0.9, 82 0.999)
Actor learning rate
le-4
Critic learning rate
le-4
Discount factor (y)
0.99
Policy update interval
Target smoothing coefficient (p)
0.005
Reward scale
0.2
Maximumnentrropy framework
Learning rate of a
3e-4
Expected entropy (H)
H. = dim(()
Deterministic policy
Exploration noise
N(0,0.13)
Off-policy
Replay buffer size
x 10c
Sample batch size
20
On-policy
Sample batch size
2000
Replay batch size
2000
driving, and industrial control. However, DACER could also enhance the exploratory capabilities and operational efficiency of military AI. potentially posing threats to citizen privacy and security.
15

---

TABLE 3
ALGORITHM HYPERPARAMETER
Parameter
Setting
Replay buffer capacity
1000000
Buffer warm-up size
30000
Buffer warm-up size (Humanoid, HalfCheetah)
200000
Batch size
256
Discount
0.99
Initial alpha ar
0.27
Target network soft-update rate p
0.005
Network update times per iteration
Action bound
[-1.
Reward scale
0.2
Hidden layers in noise prediction network
[256, 256, 256]
Hidden layers in noise prediction network (Humanoid)
[512, 512, 512]
Hidden layers in critic network
[256, 256, 256]
Activations in critic network
GeLU Mish TanhGauss
Activations in actor network
Policy act distribution Policy min log std Policy max log std Policy delay update
-20
0.5
Number of Gaussian distributions for mixing
Number of action samples in entropy estimation
200
Alpha delay update
10000
Noise scale
Noise scale À (Humanoid, HalfCheetah)
0.15
Optimizer
Adam
Actor learning rate
1 10-4
Critic learning rate
1-10
Alpha learning rate
3 10
Target entropy
0.9 dim(A)
NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: In both the abstract and the introduction we give a formulation of the contri- bution points. These include considering the reverse diffusion process novel policy approximator, proposing Gaussian mixture model method estimate the entropy of the diffusion policy and learning parameter for the regulation of the policy exploration level, and open-soureing the code. Experimental results in eight MuloCo environments as well one multimodal environment demonstrate the good performance of our method and the strong characterization capability of the diffusion policy.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
16

---

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In Section of the appendix, we acknowledge the limitations of our method- ology, estimating entropy requires a considerable amount of time, making it difficult to perfectly integrate this method with maximum entropy RL. However, in future work, we will improve the estimation efficiency by reducing the number of states used for estimation.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texx system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
17

---

Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: First, in Section 4.3, we provide pseudo-code for the algorithm I of diffusion actor-critic with entropy regulator (DACER). Second, in Section 5, we provide details of the experimental environment setup. Finally, among the two tables in the Appendix A.2, we give details of the hyperparameter configuration.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- spen-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All the reproductions of the experimental results of the baseline algorithms can obtained by running the general optimal control problem solver (GOPS). And GOPS can be searched on github. We provide the PyTorch code for DACER as function approximator in GOPS. Besides, we have implemented the DACER algorithm JAX.
Regarding the DACER algorithm, we give an implementation of all the core parts, but the complete training code we will open source after the review.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurIPS code and data submission guidelines (kttps://nips.ce cc/ oliyy) for more details.
18

---

While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. wwtpccpppplc...cc.cccccc gwwc//gg///////eee /ttggddddee///ouimiiiii /Codeeuumissiiiiieecec ssionnolccy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Appendix A.2, we give the design of all hyperparameters, including opti- mizer selection, learning rate, neural network configuration, and so on.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All results of this experiment came from in 5 random seeds. In Section 5, we mention the specific way of evaluation and the fact that all results are presented as mean standard deviation. Regarding the training curves, the solid lines represent the mean, while the shaded regions indicate the 95% confidence interval over five runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
19

---

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Section 5, we provide the CPU and GPU models used for training, which are Xeon(R) Platinum 8352V and NVIDIA GeForce RTX 4090, respectively. We take the example of training 1.5 million Humanoid-v3, and the time required to train it in the JAX framework is about hours.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https hwtpeerpp..ccppp.ccceeiiiiieeeee
Answer: [Yes]
Justification:
We made sure the code anonymous.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Appendix C, we discuss the potential positive and negative social impacts our work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate
20

---

to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We used the GOPS solver for training and cited the corresponding paper. This open-source code library is licensed under the Apache-2.0 license. Copyright @ 2022 Intelligent Driving Laboratory (iDLab). All rights reserved.
Guidelines:
existing assets. The authors should cite the original paper that produced the code package or dataset.
The answer NA means that the paper does
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information not available online, the authors are encouraged to reach out to the asset's 's creators.
21

---

13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/cood//model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "d87QsYFobN", "reviewer": "Reviewer_xVKC", "summary": "The authors propose DACER (Diffusion Actor-Critic with Entropy Regulator), an online RL algorithm that uses the reverse diffusion process as a policy in order to capture multimodal behaviours. To balance exploration and exploitation, the authors propose carrying out entropy regularization by estimating the entropy of the diffusion policy using a Gaussian mixture model. The paper demonstrates the proposed method on a set of simulated control tasks, comparing against a suite of established RL algorithms.", "questions": "See suggestions and questions in the Weaknesses section.", "limitations": "The authors should include a section on the limitations of the proposed methods (e.g. computational inefficiencies?)", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- To the best of my knowledge, the method proposed in the paper is a novel combination of new techniques – online actor-critic methods and diffusion models.\n- The paper also discusses relevant offline and online RL algorithms that make use of diffusion policies for control, clarifying the differentiating factors of this work. \n- The paper compares against a suite of well-established RL algorithms on a range of simulated control tasks (ranging from simple to complex). The experimental results show that the proposed method generally leads to improvements in learned policy performance, with additional studies showing the multimodality of learned policies and ablations on each component of the proposed method.\n- Learning multimodal policies is of significance to the RL community, and the proposed parametrization of entropy regularization for diffusion policies seems likely to be built on in subsequent work. The authors release relevant code, enabling future work to build upon this paper.", "weaknesses": "- The related works section would be strengthened by adding in references to online RL algorithms used for image generation (e.g. \"Training Diffusion Models with Reinforcement Learning\" (Black et. al, 2023), or \"DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models\" (Fan et al., 2023)). While these methods are applied in an image generation setting rather than direct control, the methods for policy improvement are related (i.e. optimizing for maximum reward with a policy based on the reverse diffusion process, learned by backpropagating through the diffusion chain), with some differences (not using actor-critic methods, learning a q-value, etc.). \n- It would be more meaningful to see Table 1 as the mean of returns in the final 10% of iterations, rather than mean of *highest* returns, the latter which may be skewed by luck/noise in the evaluation.\n- It would be interesting to see some measures of how computationally efficient the method is – since it requires stepping sequentially through the diffusion process to sample an action at each step. What are the tradeoffs?\n- Rather than method iterations, the results in Figure 1 could be better contextualized by also looking at environment steps on the x-axis (i.e. how sample efficient is each method), since the meaning of “iteration” can vary per method.\n- Minor note: The definitions of eq. (1) and (2) could be written more clearly (i.e. how do (s,a) affect the rewards on the right hand side of the equation?)\n- Minor note: for clarity, the definitions of $\\mathcal{L}_\\pi$ and $\\mathcal{L}_q$ should be stated.\n- Minor note: figures 3,4,5 should have more descriptive captions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the careful reading of our paper and constructive comments in detail.\n\n### **> Weakness 1**\nThank you for your important information. We have added a paragraph to introduce image generation with online RL algorithms. \n\n\"Diffusion models are widely applied in the field of image generation [1, 2]. Recently, several studies have employed online RL algorithms to fine-tune these models. Black et al. [3] propose an RL-based method for fine-tuning large-scale diffusion models to match potentially non-differentiable reward functions. This approach views the reverse process of diffusion models as a reinforcement learning task with time steps T, where the output of the diffusion model at each step serves both as an action and the observation for the next step. Fan et al. [4] proposed DPOK, a RLHF method for diffusion models in text-to-image generation. Given a reward model, the goal is to fine-tune the parameters of the diffusion model so that the resulting images from the sampling process achieve high rewards. The reward is a function of the final output of the entire diffusion process and, therefore, not be optimized independently across timesteps. The above work utilizes the reverse diffusion process to optimize the maximum reward, but does not learn the Q-function, as well as using the classical Actor-Critic framework.\" \n\n### **> Weakness 2**\nIt can be seen that our core comparison algorithm is DSAC. To be consistent with it, we use the highest value in 10% of the iterations. It is worth noting that each point in the experiment is the average of 10 episodes. We also set 5 different random seeds in each experiment. The results in Table 1 are the average and standard deviation of the results of the 5 random seeds. These settings avoid the influence of luck and noise to a certain extent. \n\nBut since you have mentioned it, we also give the mean and standard deviation of the 5 seed results averaged over the last 10% of iterations, **which are shown in Table 3 of the author global rebuttal 3.**\n\n### **> Weakness 3**\nAs you pointed out, the discussion on training and inference efficiency is very interesting. We acknowledge that DACER has longer training and inference time compared to DSAC. But our starting point is to **train in an offline simulation environment and then apply the trained network online**. Therefore appropriate longer training time is acceptable if performance gains can be realised (the Humanoid-v3 task with the longest training time in Mujoco took only 7 hours in JAX).\n\nWe add experiments (Task: Ant-v3) to compare the inference and training time using the JAX framework on the AMD Ryzen Threadripper 3960X 24-Core Processor, NVIDIA 3090Ti. We use the results of 10 calculations to find the mean and standard deviation. **The results are shown in Table 1 of the author global rebuttal 1.**\n\nFortunately, **DACER's 1-batch inference time is a small value within 1 ms, which is acceptable in the real-time application**. We also find that the training time of DACER is 4-5 times longer than that of DSAC. The bottleneck is the longer chain gradient of the diffusion policy. This question you presented is insightful, and it is actually what **we are working on recently**. We are going to introduce an ODE method to eliminate the bottleneck. We would advise to follow our future work. \n\n### **> Weakness 4**\nFor the sake of fairness in comparison, we maintained the same experimental setup and plotting standards as used in DSAC. In DSAC, **each iteration represents one network update, and 20 samples are sampled per iteration.** For the implementation of TRPO and PPO, we have also aligned with DSAC. In their paper, they collect 2,000 samples per iteration, with a mini-batch size of 10 and repeat_num set to 10. This setup achieves a network update corresponding to 20 samples. For env steps, you need to multiply the horizontal axis by 20.\n\n### **> Minor note 1**\nWe modify Eq.1 to $J _ \\pi=\\mathbb{E} _ {(s _ {i\\geq t},a _ {i\\geq t})\\sim\\pi}\\Big[\\sum _ {i=t}^{\\infty}\\gamma^{i-t}r(s_i, a_i)\\Big]$\n\nEq.2 to $Q(s,a)=\\mathbb{E} _ \\pi\\Big[\\sum _ {i=0}^\\infty\\gamma^ir(s_i, a_i)|s_0=s,a_0=a\\Big]$.\n\n### **> Minor note 2**\n$\\text{min}\\mathcal{L} _ {q}(\\phi) = \\mathbb{E} _ {({s},{a},{s}^{\\prime})\\sim\\mathcal{B}}\\left[\\left(r(s,a) + \\gamma \\mathbb{E} _ {s' \\sim p, a' \\sim \\pi}[Q^{\\pi}(s',a')]-Q^{\\pi} _ {\\phi}(s, a)\\right)^{2}\\right]$\n\n$\\text{min}\\mathcal{L} _ {\\pi_{}}(\\theta) = -\\mathbb{E} _ {s \\sim d _ {\\pi},a \\sim \\pi }[Q^{\\pi _ {\\mathrm{old}}}(s,a)]$\n\n### **> Minor note 3**\nThanks to your reminder, we modify the figure notes of Figure 3-5 respectively as\n\n\"**Ablation training curves for the Entropy Regulator mechanism.** DAC stands for not using the Entropy Regulator. DACER's performance on Walker2d-v3 is far better than DAC.\"\n\n\"**Ablation experiment curves for the Noise factor modulation mechanism.** Adaptive tuning of the noise factor based on the estimated entropy achieved the best performance compared to fixing the noise factor or using the adaptive tuning method with initial, end values followed by a linear decay method.\"\n\n\"**Ablation experiment curves for the different diffusion steps.** The best performance was achieved with diffusion steps equal to 20, in addition to the instability of the training process when equal to 30.\"\n\n### **> Limitation**\nIn Appendix B we discuss related Limitation and future work, which we will put in the main text in the official version.\n\n### **Reference**\n\n[1] Xie et al. \"Scalable diffusion models with transformers,\" ICCV, 2023.\n\n[2] Dhariwal et al. \"Diffusion models beat gans on image synthesis,\" NIPS, 2021.\n\n[3] Black et al. \"Training Diffusion Models with Reinforcement Learning,\" ICLR, 2024.\n\n[4] Fan et al. \"DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models,\" NIPS, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j3ao25SJQV", "reviewer": "Reviewer_NPyG", "summary": "This paper proposes using the reverse diffusion process as the policy for actor-critic-based online reinforcement learning. An EM-based mixture model is fitted to estimate current diffusion policies' entropy to balance exploration and exploitation. The proposed method, DACER, demonstrated on-par or improved performance compared with RL with deterministic and single-mode Gaussian policies in various tasks.", "questions": "1. Could the author explain why the value of PPO looks like a constant for all the states in Figure 2?  More explanations on why PPO does not work in this task would also help the understanding.", "limitations": "The computational complexity is the main limitation of this work. Although it has been mentioned in the appendix, I feel like more discussion and evaluations are needed to better understand this limitation.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. It is the first work that I know of directly using online RL to train diffusion models by directly back-propagating through the diffusion chain.\n\n2. The proposed method performed on-par or better than conventional RL methods with deterministic or single-mode Gaussian policies in some continuous control tasks.\n\n3. The writing of the paper is easy to follow, and the contributions are clearly outlined.", "weaknesses": "1. My major concern is that the calculation of entropy using GMM seems not incorrect; Eq. 15 is not equivalent to the entropy of GMM presented in Eq. 12. GMM does not have a closed-form solution for its entropy [1]. However, it could be approximated in various ways. If the author used any of the approximations, the reference should be mentioned in the paper. Furthermore, in the ablation study (Figure 4) comparing DACER with linear decay entropy, the entropy regularization does not show much performance difference. Considering the extra compute overhead by applying EM in the inner loop, I’m not convinced it is worth the effort.\n\n2. Another major concern of mine is that the proposed method seems to be very computationally heavy; for each policy update, the gradient needs to backpropagate through the entire diffusion chain. Furthermore, an inner-loop EM algorithm needs to be implemented to approximate the policy entropy. Although the training time for Humanoid-v3 is mentioned in section 5, there is a lack of comparison with other baselines implemented in the same framework (JAX) and the same hardware, which is necessary for understanding the extra computational cost.\n\n3. DACER does not show significant performance improvement over DSAC in most environments except Humanoid-v3 and Ant-v3. Given the large computational overhead, it is not sure if it is really worth using the method in given tasks.\n\n4. In Figure 1, the iteration of PPO and TRPO are reported in the number of network update steps, which is uncommon and needs proper justification, as the number of network updates does not necessarily reflect the number of simulation steps for these two algorithms.\n\n5. I am also concerned that the policy representation experiments can not fully show the representation power of diffusion policy. Diffusions are models with rich representation power as they can capture multi-modal action distribution [2] at the same state. For the task with 2d state space presented in Fig. 2, the multi-modality in action distribution only exists in the state of (0, 0), which is also why single-mode policies like DSAC and TD3 could also partially solve the task. Although the result of DACER looks slightly better than DSAC, I’m not fully convinced that this result suggests a better representation power for diffusion policy.\n\n[1] Robin S, Scrucca L. Mixture-based estimation of entropy. Computational Statistics & Data Analysis. 2023 Jan 1;177:107582.\n\n[2] Jia X, Blessing D, Jiang X, Reuss M, Donat A, Lioutikov R, Neumann G. Towards diverse behaviors: A benchmark for imitation learning with human demonstrations. arXiv preprint arXiv:2402.14606. 2024 Feb 22.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the careful reading of our paper and constructive comments in detail.\n\n### **> Weakness 1**\n1. As you pointed out, the entropy of GMM does not have a closed-form solution. Eq.15 provides **an upper bound for the entropy of GMM [1]**, which can be used for approximate estimation. We will update this reference accordingly. \n\n2. In Algorithm 1, we propose calculating **1 EM every 10,000 steps, resulting in a computational overhead that can be ignored in terms of overall efficiency.**\n\n3. The reason you did not observe a significant performance difference is that the initial and final values of the linear decay are provided by **the results of the entropy adjustment method.** We add more experimental results for different linear decay range of $\\alpha$ values. **In Figure 1 of the author global rebuttal PDF**, it can be observed that the best performance after parameter tuning only approaches that of our method, **while the computational cost of parameter tuning is significantly higher**. \n\n### **> Weakness 2**\nAs you pointed out, the discussion on training and inference efficiency is very interesting. We acknowledge that DACER has longer training and inference time compared to DSAC. But our starting point is to **train in an offline simulation environment and then apply the trained network online**. Therefore appropriate longer training time is acceptable if performance gains can be realised (the Humanoid-v3 task with the longest training time in Mujoco took only 7 hours in JAX).\n\nWe add experiments (Task: Ant-v3) to compare the inference and training time using the JAX framework on the AMD Ryzen Threadripper 3960X 24-Core Processor, NVIDIA 3090Ti. We use the results of 10 calculations to find the mean and standard deviation. **The results are shown in Table 1 of the author global rebuttal 1.**\n\nFortunately, **DACER's 1-batch inference time is a small value within 1 ms, which is acceptable in the real-time application**. We also find that the training time of DACER is 4-5 times longer than that of DSAC. The bottleneck is the longer chain gradient of the diffusion policy. This question you presented is insightful, and it is actually what **we are working on recently**. We are going to introduce an ODE method to eliminate the bottleneck. We would advise to follow our future work. \n\n### **> Weakness 3**\nBased on the above response, the appropriate longer training time relative to DACER is acceptable. As you pointed out, our performance improvements are particularly significant **in the most complex state-space tasks, Humanoid-v3 (376-dimensional state, 17-dimensional action) and Ant-v3 (111-dimensional state, 8-dimensional action), demonstrating DACER's strong potential**. In other Mujoco tasks, DSAC has already approached the performance ceiling, while DACER still achieves obvious improvements of 276 and 440 in the complex Walker2d-v3 and Hopper-v3 tasks, respectively, and outperforms or matches DSAC in all other tasks. We believe these performance gains are substantial.\n\n### **> Weakness 4**\nFor the sake of fairness in comparison, we maintained the same experimental setup and plotting standards as used in DSAC [2, 3]. In DSAC, **each iteration represents one network update, and 20 samples are sampled per iteration.** For the implementation of TRPO and PPO, we have also aligned with DSAC. In their paper, they collect 2,000 samples per iteration, with a mini-batch size of 10 and repeat_num set to 10. This setup achieves a network update corresponding to 20 samples.\n\n\n### **> Weakness 5**\nYou may have misunderstood. The multimodality of the action distribution is **not limited to the (0, 0) point**; it also exists between adjacent peaks (on two diagonals). This is why, in the results of DSAC and TD3, there are issues with flat and poorly learned regions between adjacent peaks, as described in Section 1.1. \n\nTo demonstrate the powerful multimodality of DACER, we add an experiment. We select five points requiring multimodal policies: (0.5, 0.5), (0.5, -0.5), (-0.5, -0.5), (-0.5, 0.5), and (0, 0). For each point, we sampled 100 trajectories. **The trajectories are plotted in Figure 2 of the author rebuttal PDF**. The results show that compared with DSAC, DACER exhibits strong multimodality. This also explains why only the Q-function of DACER can learn **the nearly perfectly symmetrical four peaks.**\n\nIn addition to using Multi-goal task, we use the method from [4] to **analyze the diversity of state trajectories for MuJoCo tasks, with results presented in Table 2 of the author global rebuttal 2.** This indicator can measure the algorithm's ability to explore the state space. The results show that in the Humanoid-v3, Ant-v3, Hopper-v3 and Pusher-v2 tasks, DACER achieves the highest entropy value, surpassing DSAC, SAC, DDPG, TD3, PPO, and TRPO.\n\n\n### **> Question 1**\nDuring training, we found that PPO always remains around its initial value and does not improve reward performance. \"Multi-goal\" is an environment with very sparse rewards, where huge rewards are only provided at four symmetrical peaks. In this environment, the state initialization is mostly near (0, 0). We think that PPO's conservative policy update mechanism is the fundamental reason it performs poorly in this multimodal environment.\n\n### **Reference**\n[1] M. F. Huber et al. \"On entropy approximation for Gaussian mixture random vectors,\" IEEE MFI, 2008. \n\n[2] Duan et al. \"Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation errors,\" IEEE TNNLS, 2021.\n\n[3] Duan et al. \"Dsac-t: Distributional soft actor-critic with three refinements,\" arXiv, 2310.\n\n[4] Xiao et al. \"Multi-Style Distributional Soft Actor-Critic: Learning a Unified Policy for Diverse Control Behaviors,\" in IEEE TIV, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mi2RWhhE1r", "reviewer": "Reviewer_3PmU", "summary": "The paper presents a novel method using diffusion models as a policy parameterization for online reinforcement learning. The method works by learning a Q function and backpropagating through the reverse diffusion process in order to update the diffusion policy weights, similarly with Diffusion-QL. In order to aid exploration, the authors estimate the entropy of the policy and use that to update $\\alpha$, which is used to adjust the noise added to actions after sampling.", "questions": "Random questions and comments\n 1. Section 2 paragraph 2, you frequently cite incompatibility / combinability with past methods or a lack thereof as an important feature of an algorithm, using this metric to claim a victory over other methods using diffusion for online. I disagree that this is a useful metric for evaluating algorithms and was wondering whether you could elaborate on the logic behind this claim.\n 2. Line 108 you claim that DIPO and QSM perform worse than SAC and thus you don't compare to them. However, both papers claim to outperform SAC so I don't see why you believe they perform worse. \n 3. The method seems to rely on the fact that a GMM can accurately fit your action distribution. Is there a reason you can't simply use that GMM for your policy? Seems to me that that would be more efficient and easy to implement. I think this is an important ablation to run to show that the full expressive power of the diffusion policy is actually useful in the settings you consider.\n 4. Figures 3-5 all have the same caption, \"Trainining curves\" [sic]. The caption should fully describe what is going on in the figure.\n 5. Your method takes 7 hours to train with the JAX implementation and apparently 4-5 times longer with PyTorch. How long do the baseline algorithms take to train? Assuming that they train for much less time, can you make a case that the extra time is worth it for generally modest performance improvements?\n 6. In Figure 3 what is DAC (I assume that this is the ablated version but you never state the meaning of that acronym)? It seems to outperform your method. Is the legend switched?\n 7. One of the key claims of your paper is that a diffusion policy is necessary due to its ability to accurately model multimodal distributions, and yet your main evaluation suite doesn't include any tasks that require this. You do have a toy problem where this is necessary, but I believe that additional experiments in an environment such as Push-T from Cheng et al. would help you to provide evidence for this claim.\n 8. Why do your plots show learning over iterations rather than the more common metric of the number of environment timesteps?\n 9. The core novelty of your method is using the GMM for adaptive noise injection, but I don't fully understand whether this is necessary. It would be nice to see how the algorithm performs with a wider range of fixed $\\alpha$ values with and without linear decay, across several environments. If you still show a win over all of these it shows that the adaptive noise injection is necessary. Even if you tie with a few, you can still use this experiment to argue that the adaptive noise injection helps you to avoid an expensive hyperparameter tuning process. \n 10. $\\alpha$ is overloaded. You use it as a variable in your mathematical description of the denoising process as well as the variable used to inject noise into the policy.\n 11. You argue that the diffusion model is important because it enables multimodal policies, and yet at the end of the day you add unimodal noise to add exploration. Have you looked into whether this prevents the policy from learning multimodal behaviors?\n 12. How did you tune the hyperparameters for your algorithm and for the comparisons?\n\nNit picks:\n - The text in all of the plots is far too small\n - The training curves would be much easier to read if the curves were smoothed.\n - The caption of Table 1 written entirely in capital letters is hard to read.\n - It is almost impossible to distinguish PPO from DDPG in your plots.", "limitations": "The authors discuss the limitations of their work, although they do it in the appendix.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper's initial results are strong and I think with further experimentation it could be a strong paper\n - Few papers have studied the application of diffusion policies to online RL, and this paper presents a fairly straight-forward method in that direction which seems to achieve strong performance compared to baselines.\n - The presentation of the method is generally easy to understand\n - The use of GMM to estimate the entropy of the diffusion policy is a novel and creative trick", "weaknesses": "- A glaring issue with the paper is that the authors chose not to compare to any other papers using diffusion policies for online RL. They claim without citation that these papers perform worse than SAC although these papers claim to outperform SAC. In order to fully contextualize this paper I feel that these comparisons are essential.\n - As I discuss in the following section, several key results are missing.\n - The presentation is generally pretty sloppy as I discuss at several points in the following section.\n - I worry that the method may be very slow, and that the gains in performance may not be enough to justify the added computational cost. It would be useful to see some data about the comparative wall clock times of the algorithm and baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for insightful review. We report below the changes we are currently making to address your comments.\n# Q1\nOur work emphasizes proposing the reverse diffusion process as a novel policy expression, combinable with existing online RL algorithms. Section 2.2 highlights the method's portability. To prove it, **global rebuttal Fig 4 includes extra experiments combining the diffusion policy with SAC.**\n\nHowever, we feel that your suggestion is good and we modifiy the relevant statement in Section 2.2 to  \n\n\"Yang et al.[1] pioneered this approach by proposing action gradient method. This approach achieves policy improvement by updating the action in the replay buffer through the $\\nabla_a Q\n$, followed by mimicry learning of the action post-update using a diffusion model. However, action gradient increased the additional training time. Furthermore, it is difficult to fully learn both the action gradient and imitation learning steps simultaneously, which also resulted in suboptimal performance of this method in MuJoCo tasks. Penka et al.[2] proposes Q-score matching (QSM), a new methodology for off-policy reinforcement learning that leverages the score-based structure of diffusion model policies to align with the $\\nabla_a Q\n$. This approach aims to overcome the limitations of simple behavior cloning in actor-critic settings by integrating the policy's score with the Q-function's action gradient. However, QSM needs to accurately learn  $\\nabla_a Q\n$ in most of the action space to achieve optimal guidance. This is difficult to accomplish, resulting in suboptimal performance of QSM.\"\n# Q2\nAs you mentioned, comparing the performance with DIPO and QSM is meaningful. In fact, we have previously conducted relevant experiments. However, using the hyperparameters provided in their papers and testing within our JAX framework, we found that their performance did not surpass SAC as claimed. To avoid any misunderstanding that we intentionally lowered their performance, we chose not to include the experimental results. To address your concerns, **we have added the results in the global rebuttal PDF Figure 3**. We will proactively contact the authors to discuss why the performance does not exceed SAC.\n# Q3\nFirst, using GMM as a policy requires pre-specifying the number of Gaussian components, which limits its ability to approximate any continuous distribution. Second, in DACER, entropy estimation updates the noise parameter $\\alpha$, where some estimation error is acceptable. Finally, we conduct experiments replacing the diffusion policy with GMM (3 Gaussian components) in MuJoCo tasks, showing that **performance is lower than that of the diffusion policy, as illustrated in Figure 4 of the global rebuttal PDF**.\n# Q4\nThank you for your reminder. We have written the revisions for **the titles of Figures 3-5 in global rebuttal 4**.\n# Q5\nYour concern about training time is understandable, but our starting point is to **train in an offline simulation environment and then apply the trained network online**. Therefore appropriate longer training time is acceptable if performance gains can be realised.\n\nWe add some comparative  experiments and **the results are shown in Table 1 of the author global rebuttal**.\n\nFortunately, **DACER's 1-batch inference time is a small value within 1 ms, which is acceptable in the real-time application**. We find DACER's training time is 4-5 times longer than DSAC due to the longer chain gradient of the diffusion policy. Your question is insightful and **aligns with our recent work**. We plan to introduce an ODE method to eliminate this bottleneck.\n# Q6\nThank you for your careful discovery that the DAC and DACER legends were written backwards here, which we have corrected.\n# Q7\nAs you said, it's important to have an experiment that clearly characterizes the need for a multi-peak policy. We choose the vehicle bifurcation experimental task presented in [3] (like push-T) and plot  **the action distribution at the bifurcation point in Figure 5 of global rebuttal PDF**. \n# Q8\nFor the sake of fairness in comparison, we maintained the same experimental setup and plotting standards as used in DSAC. In DSAC, **each iteration represents one network update, and 20 samples are sampled per iteration.** \n# Q9\nYour proposal is strong. To show that adaptive noise injection can achieve optimal performance without parameter tuning, we conducted experiments in three environments, **with training curves in Figure 1 of the global rebuttal PDF**. The results indicate that while certain linear decay results can perform comparably to adaptive noise injection, most of the results are not as good as adaptive noise injection.\n# Q10\nThank you for pointing out that we will modify $\\alpha$ in the mathematical description of the denoising process to $\\omega$.\n# Q11&12\n**It does not prevent policy from learning multimodal behavior**. First, the diffusion policy's multimodality comes from the reverse diffusion action generation, with added noise just being a little perturbation. Second, experiments show the distance between peaks is generally greater than 0.5, while the injected noise magnitude is within 0.272, decaying to within 0.15 as training progresses.\n\nFor the second problem, we used hyperparameters from the DSAC paper for comparison algorithms. For DACER, the diffusion steps, $\\lambda$, and $\\alpha$ learning rate were consistently set at 20, 0.1, and 0.03 for nearly all experiments.\n# Picks\nThanks to your suggestions, we will zoom in on the text in the image, add smoothing to the training curves (all curves in global rebuttle have been added), change the title of Table 1 to lowercase.\n\n### **Reference**\n[1] Yang et al. \"Policy representation via diffusion probability model for reinforcement learning,\" arXiv, 2305.\n\n[2] Penka et al. \"Learning a diffusion model policy from rewards via q-score matching,\" arXiv, 2312.\n\n[3] Zou et al. \" Policy Bifurcation in Safe Reinforcement Learning,\" arXiv, 2403.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FYLxMbLerj", "reviewer": "Reviewer_QFqf", "summary": "The paper proposes DACER — an actor-critic that uses the inverse diffusion process as policy. Additionally, some noise is added to actions to increase the entropy (similar motivation as in SAC but implemented differently). Empirical evaluations and ablations show that the method is quite capable.", "questions": "The proposed entropy regulation scheme seems a bit unusual. Did you try other options (e.g. like in TRPO)?", "limitations": "Please add a paragraph discussing the limitations", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Originality: good. The paper proposes a novel method how to use diffusion as policy representation for a SAC-like algorithm.\n\nQuality: good. The method is presented and evaluated thoroughly.\n\nClarity: excellent. The paper is easy to follow.\n\nSignificance: good. The method will be of interest to the RL community.", "weaknesses": "1) No comparison to an algorithm with multimodal policy. The key advantage of the diffusion policy compared to the usual Gaussian is its multimodality. But there are other algorithms with multimodal policies, e.g., [Reinforcement Learning with Deep Energy-Based Policies](http://proceedings.mlr.press/v70/haarnoja17a.html?ref=https://githubhelp.com) and many newer works (check \"Cited by\" on that paper).\n2) No clear demonstration where multimodality helps. Some toy task or better not-toy task where one sees a clear advantage would be nice to see.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate you for the careful reading of our paper and detailed discussions.\n\n### **> Weakness 1**\nAs you mentioned, it is meaningful to compare with other algorithms that have multimodal characteristics. We chose to compare the performance with two online RL with Diffusion model algorithms mentioned in the related work (DIPO and QSM). **In Figure 3 of the author global rebuttal PDF, it can be seen that DACER achieved the best performance**.\n\n### **> Weakness 2**\nWe answer this question in two ways. First, **some not-toy tasks are inherently multimodal**. Zou et al.[1] proposes a vehicle control scenario in which the policies are multimodal, i.e., in the case where traffic regulations are not taken into account and the ego car is directly behind the front vehicle and there is only one weekly vehicle, it is an optimal policy for the ego car to overtake both from the left and from the right. In this case, **only a multimodal policy is able to deal with the situation**. We add experiments comparing the action (steering wheel angle) distributions of DSAC and DACER at the bifurcation point. We use the diffusion policy to sample 200 different actions and **plot a histogram in Figure 5 of the author global rebuttal PDF**. The results indicate that **DACER learns a bimodal policy**, whereas DSAC can only learn a unimodal distribution for overtaking from one side.\n\n Second, we think **multimodality enhances exploration**. To assess the multi-style capability of DACER, we use the method proposed in [2]. This method analyze the diversity of state trajectories by computing the information entropy from 1,100 independent simulation episodes. The experimental results are plotted in **the author global rebuttal Table 2**. The results show that in the Humanoid-v3, Ant-v3, Hopper-v3 and Pusher-v2 tasks, DACER achieves the highest entropy value, surpassing DSAC, SAC, DDPG, TD3, PPO, and TRPO.\n \n\n### **> Question 1**\nThis question of yours is one of the most creative points of DACER. The entropy of the diffusion policy is difficult to calculate analytically, which means we cannot use entropy regularization methods like in TRPO to adjust the entropy (the gradient cannot be backpropagated to the diffusion policy). Additionally, we tried to adaptively adjust the single-step denoising coefficient based on the estimated entropy during the reverse diffusion process, but the results were not as good as the current methods.\n\n### **> Limitation**\nIn Appendix B we discuss related Limitation and future work, which we will put in the main text in the official version.\n\n### **Reference**\n\n[1] Zou et al. \" Policy Bifurcation in Safe Reinforcement Learning,\" arXiv, 2403.\n\n[2] Xiao et al. \"Multi-Style Distributional Soft Actor-Critic: Learning a Unified Policy for Diverse Control Behaviors,\" in IEEE TIV, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Overall author rebuttal:

We thank all reviewers for their thoughtful comments. We greatly appreciate all the reviewers' acknowledgment that our method **is very novel and is the cutting-edge work combining diffusion model and online RL. In addition, our method achieved SOTA in MuJoCo tasks and is very easy to follow**. To address your concerns about the training and inference time, the lack of comparative results with DIPO and QSM, the wide range of noise factor ablation experiments, the performance of using GMM directly as a policy, and how multimodality is represented and its importance, we add new experiments and evaluations:

### **1. Experiments with training and inference time**
We add experiments (Task: Ant-v3) to compare the inference and training time using the JAX framework on the AMD Ryzen Threadripper 3960X 24-Core Processor, NVIDIA 3090Ti. We use the results of 10 calculations to find the mean and standard deviation. Inference time represents the time required for the policy to generate actions. Backward time represents the time required for gradient propagation. (It should be noted that the backward time of DIPO consists of two parts: gradient propagation and action gradient. Besides, the diffusion steps of DACER and DIPO are both 20, and the action gradient steps of DIPO are 20.)

**Table 1 Time statistics results.** Under the same hardware conditions, the inference and backward time of the three algorithm policy functions DACER, DIPO and DSAC are compared in the Ant-v3 task with different batches.

| Mode            | Batch | DACER                     | DIPO                                          | DSAC                      |
|-----------------|-------|---------------------------|-----------------------------------------------|---------------------------|
| Inference time  | 1     | 0.472ms ± 0.006ms         | 0.502ms ± 0.011ms                             | 0.188ms ± 0.027ms         |
| Inference time  | 100   | 0.968ms ± 0.004ms         | 0.970ms ± 0.006ms                             | 0.190ms ± 0.020ms         |
| Backward time   | 1     | 1.283ms ± 0.036ms         | 0.582ms ± 0.020ms (gradient propagation); 0.801ms ± 0.008ms (action gradient) | 0.571ms ± 0.038ms         |
| Backward time   | 100   | 2.428ms ± 0.028ms         | 0.601ms ± 0.021ms (gradient propagation); 1.483ms ± 0.006ms (action gradient) | 0.602ms ± 0.015ms         |


### **2. Experiment on evaluating the performance of algorithm exploration.**
To assess the multi-style capability of DACER, we use the method proposed in [1]. This method analyze the diversity of state trajectories by computing the information entropy from 1,100 independent simulation episodes.

**Table 2 Information entropy results.** The results show that in the Humanoid-v3, Ant-v3, and Hopper-v3 tasks. DACER achieves the highest entropy value.

| Mode        | DACER  | DSAC   | SAC    | DDPG   | TD3    | PPO    | TRPO   |
| ----------- | ------ | ------ | ------ | ------ | ------ | ------ | ------ |
| Humanoid-v3 | 16.743 | 16.711 | 16.668 | 16.693 | 16.284 | 14.699 | 14.721 |
| Ant-v3      | 16.821 | 16.716 | 16.683 | 16.609 | 15.376 | 16.307 | 16.339 |
| Hopper-v3   | 12.453 | 11.822 | 9.559  | 10.462 | 10.141 | 10.803 | 10.310 |
| Pusher-v2   | 10.722 | 9.719  | 10.216 | 9.471  | 8.286  | 9.714  | 9.632  |




### **3. Average return of the last 10% of iterations averaged over the 5 seeds.**

**Table 3  Average final return.** Computed as the **mean** of the average return values observed in the final 10% of iteration steps per run, with an evaluation interval of 15,000 iterations. The mean value for each task is bolded. ± corresponds to standard deviation over five runs.

| Task|DACER|DSAC|SAC|TD3|DDPG|TRPO|PPO|
| ------------------------- | ----------- | ----------- | ----------- | ----------- | ------------ | ----------- | ----------- |
| Humanoid-v3| 10896 ± 306 | 9630 ± 469  | 7515 ± 399  | 5305 ± 277  | 3558 ± 848   | 744 ± 333   | 5398 ± 2211 |
| Ant-v3| 7584 ± 211  | 6643 ± 324  | 6208 ± 1412 | 5772 ± 386  | 3474 ± 784   | 5851 ± 537  | 5908 ± 247  |
| Halfcheetah-v3| 16489 ± 276 | 16182 ± 779 | 15884 ± 355 | 7775 ± 3627 | 11710 ± 3108 | 4586 ± 1007 | 5503 ± 2084 |
| Walker2d-v3| 6496 ± 134  | 6171 ± 301  | 5364 ± 817  | 4785 ± 104  | 2577 ± 266   | 4708 ± 732  | 3634 ± 573  |
| Inverteddoublependulum-v3 | 9360 ± 0    | 9360 ± 0    | 9360 ± 0    | 9311 ± 11   | 8983 ± 11    | 5852 ± 2335 | 9355 ± 2    |
| Hopper-v3| 3679 ± 362  | 3111 ± 616  | 2147 ± 621  | 2909 ± 510  | 1850 ± 275   | 2805 ± 404  | 1932 ± 290  |
| Pusher-v2| -20 ± 0     | -21 ± 1     | -23 ± 2     | -25 ± 4     | -37 ± 2      | -26 ± 5     | -24 ± 1     |
| Swimmer-v3| 146 ± 2     | 135 ± 6     | 135 ± 14    | 114 ± 26    | 140 ± 4      | 69 ± 38     | 129 ± 1     |


### **4. Changes to the title of Figure 3-5.**
"**Ablation training curves for the Entropy Regulator mechanism.** DAC stands for not using the Entropy Regulator. DACER's performance on Walker2d-v3 is far better than DAC."

"**Ablation experiment curves for the Noise factor modulation mechanism.** Adaptive tuning of the noise factor based on the estimated entropy achieved the best performance compared to fixing the noise factor or using the adaptive tuning method with initial, end values followed by a linear decay method."

"**Ablation experiment curves for the different diffusion steps.** The best performance was achieved with diffusion steps equal to 20, in addition to the instability of the training process when equal to 30."

### **Reference**

[1] Xiao et al. "Multi-Style Distributional Soft Actor-Critic: Learning a Unified Policy for Diverse Control Behaviors," in IEEE TIV, 2024.

Meta-review: This paper proposes a novel method for online actor-critic and diffusion models. 

Citing the reviewer xVKC, "...The experiment results show that this method leads to improvements in learned policy performance, with additional studies showing the multimodality of learned policies and ablations on each component of the proposed method. Learning multimodal policies is of significance to the RL community...". 

This method might be of interest for the research community, so this paper is recommended to be accepted to NeurIPS24 for a poster presentation.
Decision: Accept (poster)