Generalizing Consistency Policy to Visual RL with Prioritized Proximal Experience Regularization
Authors: Haoran Li
Zhennan Jiang
YUHUI CHEN
Dongbin Zhao
Venue: NeurIPS 2024 poster (2024)
Keywords: Visual Reinforcement Learning
Reinforcement Learning
Consistency Model
Dormant Neuron Phenomenon
Diffusion Model
Primary area: reinforcement_learning
TL;DR: This paper extends consistency policy to visual RL and achieves new SOTA performance on 21 visual control tasks.

Abstract: With high-dimensional state spaces, visual reinforcement learning (RL) faces significant challenges in exploitation and exploration, resulting in low sample efficiency and training stability. As a time-efficient diffusion model, although consistency models have been validated in online state-based RL, it is still an open question whether it can be extended to visual RL. In this paper, we investigate the impact of non-stationary distribution and the actor-critic framework on consistency policy in online RL, and find that consistency policy was unstable during the training, especially in visual RL with the high-dimensional state space. To this end, we suggest sample-based entropy regularization to stabilize the policy training, and propose a consistency policy with prioritized proximal experience regularization (CP3ER) to improve sample efficiency. CP3ER achieves new state-of-the-art (SOTA) performance in 21 tasks across DeepMind control suite and Meta-world. To our knowledge, CP3ER is the first method to apply diffusion/consistency models to visual RL and demonstrates the potential of consistency models in visual RL.

Generalizing Consistency Policy to Visual RL with Prioritized Proximal Experience Regularization
Haoran Li
Zhennan Jiang Institute of Automation, Chinese Academy of Sciences
Institute of Automation, Chinese Academy of Sciences
University of Chinese Academy of Sciences lihaoran2015@ia ac cn
University of Chinese Academy of Sciences
jiangzhemaan20248ia. ac. cn
Yuhui Chen
Dongbin Zhao"
Institute of Automation, Chinese Academy of Sciences
Institute of Automation, Chinese Academy of Sciences
University of Chinese Academy of Sciences chenyuhui202200.. ac cn
University of Chinese Academy of Sciences dongbin. caao0i. a ac cn
Abstract
With high-dimensional state spaces, visual reinforcement learning (RL) faces significant challenges in exploitation and exploration, resulting in low sample efficiency and training stability. As a time-efficient diffusion model, although consistency models have been validated in online base- RL, it is still open question whether it can be extended to visual RL. In this paper, we investigate the impact of non-stationary distribution and the actor-critic framework on consistency policy in online RL, and find that consistency policy was unstable during the training, especially in visual RL with the high-dimenssonal state space. To this end, we suggest sample-based entropy regularization to stabilize the policy training, and propose a consistency policy with prioritized proximal experience regularization (CP3ER) to improve sample efficiency. CP3ER achieves new state-of-the-un (SOTA) performance in 21 tasks across DeepMind control suite and worad. To the best of our knowledge, CP3ER is the first method to apply diffussion/cosiitencyy models to visual and demonstrates the potential of consistency models visual RL. Our project page is hosted at htttttts//ssmieeie . github. io/ PPCCCP3RRPaae
Introduction
RL has achieved remarkable results in many fields, such video games 11 Go 2 Chess 3 4 and robotics [5+8]. Since it is hard to parameterize the complex policy distribution over high-dimensional state and continuous action spaces, the performance and stability of visual RL are still unsatisfactory. As the most common policy distribution, Gaussian distribution is easy to sample, but its unimodal nature limits the expressiveness to represent complex behaviors 9 While complex distributions have the rich, expressive power to improve the exploration ability [10], the difficulty of sampling makes it hard to apply to online RL directly. Parameterizing the complex policy distribution to balance ease of sampling and expressiveness is bottleneck to improving the efficiency of visual RL.
As emerging generation model, the diffusion model [1I stands out in fields such image generation 12 and video generation [6] with its ability to model complex distributions and ease of sampling characteristies. These properties have also been explored for learning complex
"Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

policy 17]. For example, diffusion models are used to imitat the diverse expert policies [8 19 or trajectories |0 in datasets. In addition, due to their excellent expressive and data generation abilities, diffusion models are often employed to address policy constraints (64 and data scarcity
21 27 28 in offline RI., Most of these applications are limited to offline learning due to the demand
for pre-collected datasets to train diffusion models.
Applying diffusion models in online RL will face different problems than offline RL. Firstly, unlike pre-collected data in offline RL, the data distribution in online RL is non-stationary [29]. and it is currently unclear whether this change will impact training diffusion models. Secondly, since the optimal policy distribution is unknown, samples from this distribution are inaccessible, resulting in the ill- ill-posed traditional score matching problem [30]. In addition, the ineffiiiccc of diffusion models [31] will become more prominent with large number of online interactions, leading to unacceptable time costs for online learning. As an efficient diffusion model, the consistency model 32 directly establishes a mapping from noise to denoised data, which is employed for online RI. and achieves time efficiency and better performance 33 These methods simply replace the Gaussian model in the actor-critic framework with the consistency model and train consistency policy with the Q-loss. Although they achieve competitive performance in state-based RL tasks, this training method is incompatible with traditional score matching for diffusion models. Therefore, the question is whether this training framework is suitable for consistency model- based policy training, especially for visual RL tasks with digeensinii state spaces.
In this paper, we investigate the impact of -on-stationayy dataset and the actor-critic framework on consistency policy. By analyzing the dormant ratio [29] of the policy network, we find that the non-stationary of training data is not the main factor affecting the instability of consistency policy, while the Q-loss in the actor-critic framework leads to a sharp increase in the dormant ratio of the policy network, resulting in the loss of complex expression ability, which is particularly significant in visual RL tasks. To address the above issues, we suggest sample-based entropy regularization to stabilize the policy training and propose the prioritized proximal experience regularization, which uses weighted sampling to construct an appropriate proxy policy for policy regularization and achieves consistency policy. Overall, our contributions are as follows:
We investigate the impact of non-stationary distribution and actor-critic framework on consistency policy in online RL, and find that the Q-loss of the actor-critic framework can impair the expressive ability of the consistency model, leading to unstable policy training. This phenomenon is particularly significant in visual RL tasks.
We suggest sample-based entropy regularization and propose consistency policy with prioritized proximal experience regularization (CP3ER) which significantly enhances the stability of policy training with the Q-loss under the actor-critic framework.
Our proposed method performs new SOTA in 21 visual control tasks, including DeepMind control suite and Meta-world tasks. To our knowledge, our proposed CP3ER the first method to apply diffussion/comsistencyy models to visual RL.
2 Related Work
2.1 Diffusion Model in Reinforcement Learning
Due to its high-quality sample generation ability and training stability, diffusion models [1] have been widely applied in fields such as image generation, video generation, and natural language processing and have also been promoted in RI. Since the diffusion model can represent complex distribution
in datasets, it is commonly used in offline RL to model behavior policies [18] 25 35) or expected
policies [34] 38 to meet the requirements of diversity policy constraints and achieve a balance
between constraint and exploitation. The diffusion model can also model trajectory distribution
20 39 40, achieving specified trajectory generation under different guidance. In addition, diffusion
models are also employed to generate data to augment limited training data [7]28 28
Although diffusion models have been widely applied in offline learning, using diffusion models in online learning remains a challenging problem. [41] proposes the concept of action gradient, which uses value function to estimate the gradient of actions and updates the actions in the replay buffer. The diffusion model-based policy is trained based on the updated actions. [42) employs a diffusion
model the world model to generate complete rollouts at once instead of auto-regressive generation.

---

30 introduces the Q-score matching (QSM), which iteratively matches the parameterized score of policy with the action gradient of its Q-function. Considering the low inference efficiency and long training time of diffusion models in RI training, 33 and [4 use consistency models instead of diffusion models and implement policy training under the actor-critic framework, achieving excellent performance in continuous control tasks.
2.2 Visual Reinforcement Learning
Compared to state-based RL, visual RL is faced with nigh-dimensional state space and continuous action space and is sensitive to training parameters and random seeds, which leads to unstable training and sample inefficiency. Image data augmentation 44 45 is a common technique to alleviate the above problems. In addition, auxilliary losses are usually combined to improve the efficiency of state representation learning from the image, such as contrastive learning loss [46], state representation loss [4 48], action and state representation loss [49]. and self-supervised Ioss [50]. Recent works have focused on enhancing the stability of visual RL from micro perspective of neural networks. For example, [5I) proposes the visual dead trial phenomenon and introduces adaptive regularization method for convolutional features. [52] proposes the concept of dormant neuron phenomenon to explain the behavior of the policy network during RL training. [53] controls the dormant ratio of the policy network during training so that it achieves the SOTA performance on multiple tasks.
3 Preliminary
3.1 Reinforcement Learning
Online RL solves sequential decision problems, typically modeled through Markov Decision Pro- cesses (MDP). MDP is represented by 6 tuples (S, R. T, Po.7). Here, S the state space. is the action space, R and T represent the reward function and state transition function of the environment, respectively. Pa is the initial distribution of the state, and y is the discount factor. In visual RI., is difficult for agents to directly obtain the state 8, from the image E O, where O is observation space. Therefore, an image encoder f(-) is usually required, and the state estimated from the image through this encoder. The goal of the agent is to learn an optimal policy x² and the corresponding encoder f to maximize the expected cumulative reward E (x((--) 12 7 7/rr] under that policy.
3.2 Consistency Policy
The consistency model 132 is a new diffusion model proposed to address the time inefficiency caused by hundreds of reverse diffusion steps in diffusion models. It replaces the iterative denoising process with learned score functions in traditional diffusion models by constructing a mapping between noise and denoised samples, and directly maps any point on the probability flow ordinary differential equation (ODE) trajectory to the original data in the reverse diffusion process. Thus, only requires a small number of steps or even one step to achieve the generation from noise to denoised data. Consistency policy B33 34] is new policy representation under the actor-critic framework, which replaces traditional Gaussian models with the consistency model and updates the policy by maximizing the state-action value. Consistency policy is defined
(a(a,) st) _ Caksp(Tx)aa + Come(Te) Fa :Talst)
(1)
where (Tk kk E [N]} sub-sequence of time points the time period [c, K] with T1 and TN K. at is the noised action and a²" = a, + Tx2 where z ~ NN T) is Gaussian noise. Fg is a trainable network that takes the state 8t condition and outputs an action of the same dimension as the input a/. (-) and Cour(-) (-) are differentiable functions such that Cakip(e) = 1 and Ceua(e) = 0 to ensure consistency policy differentiable at ты e is real number close to 0. To train this policy, (33) directly applies the above policy to the actor-critic framework and updates the policy using the following the Q-loss, which is named Consistency-AC.
(0)
Eg,,8,ar<,, (st, a:)]
where B is the replay buffer and Q is the state-action value function. Compared to diffusion- based policies, consistency policy have significant advantages inference speed and performance online learning tasks [33].
3

---

3.3 Dormant Ratio of Neural Networks
The expressive ability is crucial for training the policy with RL. 29
proposes the concept of dormant ratio 8r, which quantifies the expression ability of a neural network by calculating the proportion of dormant neurons in the neural networks.
Σ, H N,
B.
(3)
where N' represents the number of neurons in the I-th layer. 1 is the number of neurons in the (-th layer whose score of is less than т. The score of each neuron is calculated as follows:
EEDD (x)
s 1 NO Eee B (EED)L(())
Here (.(()) is the actívation function of the i-th neuron in the l-th layer. 'D is the distribution of the input x. In the following sections of this paper, we use the dormant ratio to evaluate the expression ability of consistency policy during the training.
As introduced in [29 the dormant ratio of neural network indicates the proportion of inactive neurons and is typically used to measure the activity of the network. A higher dormant ratio implies fewer active neurons in the network, implying the network's capacity and expressiveness are damaged. In RL, the episode return is closely related to the dormant ratio of the policy network. A higher dormant ratio results in more lazy action outputs, inactive agent behavior, and lower episode returns; conversely, when policy performance is good, the policy network is usually more active, and the dormant ratio is typically lower. This phenomenon has been reported in 29 53-55
Is Consistency-AC Applicable to Visual RL?
Does the non-stationary distribution in online RI affect the train- ing of consistency models? Unlike offline RL, online RL does not have pre-collected datasets. The data distribution for training the policy is constantly changing with policy improvement. So, whether
Halfcheetah
0131
cttre
this non-stationarity distribution affects the training of consistency models is a question that needs to be explored. In order to investigate the impact of non-stationarity of data for consistency model training, we record the dormant ratio of the policy network during consis- tency model training under two different settings: online training and offline training. We employ two tasks (MuJoCo Halfcheetah and MuJoCo Walker2d) and conduct 4 random seeds for each setting. In order to eliminate the impact of Q-loss, we follow the behavior clone setting and train the consistency model with consistency loss [32]
Dorment
Gradient Steps
Walker2d
cttre
using data from offline datasets or online replay buffers. The distribu- tion of the data in the replay buffer varies with policy improvement. The results are shown in Figure [] Although there is a difference in the dormant ratios between online and offline learning settings in the Halfcheetah task, the overall trend is the same. We speculate that this difference is caused by the diversity of the samples included the dataset. For the Walker2d task, the dormant ratios are nearly the same under two different settings. Therefore, we can infer that the non-stationary distribution of online RL does not significantly affect the consistency model training.
Dermen
U000 2000 20000 2050
Steps
Figure 1: The dormant ratios of the policy under the online and offline training.
Is the actor-critic framework suitable for training consistency policy? The actor-critic frame- work is a highly effective policy training framework for online RL, in which the policy network achieves policy improvement by maximizing the value function. Some works [33 34] directly apply consistency models to this framework. Although they achieve good results in RL tasks with low dimensional state spaces, whether the actor-critic training framework is compatible with consistency model training remains a question that needs further investigation. To evaluate the impact of the actor-critic framework on the training of consistency models, we compare the dormant ratios of policy

---

Jenta
Dommer faato
Dommar Notro
Mycclage Rusumm
Demant Batoo
Aeuraage Rulumn
Demoan Rotto
Aeeraggh
Gradient Steps
Gradint Sheps
Gracient
Stepe
Gradint Stepe
(a) Online Halfcheetah
(b) Online Walker2d
(c) Visual Cheetah-run
(d) Visual Walk
Figure 2: The dormant ratios of the policy networks with different losses and observations.
networks under the consistency loss and Q-loss settings under the actor-critic framework. The results are shown in (a) and (b) of Figure 2 the solid line shows the dormant ratio of the policy while the dashed line shows the performance of the policy. When training the policy with the consistency loss, the dormant ratios of the network show a trend of first decreasing and then increasing. This means that the policy learns the distribution from the data and then overfits the distribution. When training the policy with the Q-loss, the dormant ratio of the policy network will rapidly increase and maintain high value, which means that the policy network will quickly fall into local optima, making the policy no longer change. In addition, we can also see that when using the Q-loss, the variance of the dormant ratios is relatively large under different random seeds. When the dormant ratio is low, the policy network can iterate properly to learn good policy. Therefore, we can determine that the Q-loss under the actor-critic framework will destabilize the consistency policy training.
Will high- nigh-dimensional state space exacerbate the degradation phenomenon of consistency policy? Compared to RL with low dimensional state space, training stability in visual RL is still challenge. In order to investigate whether the degradation phenomenon of consistency policy will become more significant under visual RL tasks, we compare the dormant ratios of the policy networks with the state as input and image as input on tasks (Walker-walk and Cheetah-run in DeepMind control suite) under the setting of online learning. During the training process, only the Q-loss was used. To maintain consistency in the settings, we only count the dormant ratio of the multilayer perceptron (MLP) of the policy newtork in the image-based settings. The results are shown in (c) and (d) of Figure 2 Similar to using the state as input, in visual RL with the image as input, most of the neurons in the MLP of the policy network go dormant. Unlike the high variance of the former, the dormant ratios of consistency policy network in visual RL maintain low variance and a high value. This indicates that there have been almost no successful trials under different random seeds. Therefore, we can infer that visual RL will exacerbate the instability of consistency policy training caused by the Q-loss under the actor-critic framework.
5 Consistency Policy with Prioritized Proximal Experience Regularization
Consistency Policy with Entropy Regularization. To solve the problem of consistency policy quickly falling into local optima under the influence of the Q-loss, we introduce policy regularization to stabilize policy improvement. Here, we employ entropy regularization to constrain policy behavior. The objective of RL
DC
((0)= Es Ba
r're(s;: at)l n, [log To [s]]
(5)
where Ty is the proxy distribution required for policy regularization. Entropy regularization is a commonly method for stabilizing policy training in RL. When n. is uniform distribution, the above objective is equal to maximum entropy RL, which maximizes the entropy of the policy while optimizing the return. The prerequisite for this method is to obtain the closed form of the policy distribution to calculate its entropy. However, for diffusion models or consistency models, obtaining the closed form of the policy distribution is very difficult. Thanks to the development of generative models, we can use score matching instead of solving analytic entropy in entropy regularization RL, thus achieving sample-based policy regularization. Therefore, the training loss of consistency policy with the entropy regularization is:
C, (0) -E QQ az)l + (0)
(6)
5

---

O
at
Environment
we
Gritic
Replay
Augmentation
Mixture Gaussian
Policy
CP3ER
42
(a)
(b)
Figure 3: (a) The framework of CP3ER, where PPE is the abbreviation of prioritized proximal experience. (b) The sampling weights 3 with different
where Le consistency loss defined by following:
(e) Bxp.:/i..N--Deeeeeeessss-ee (TTE)) a -.411,, To(s,, mk))
Here X()) is step-dependent weight function, :) is distance metric. Since there is need to obtain the closed form of the proxy distribution, only the data under that distribution needs to be obtained, making the selection of proxy distribution flexible. The remaining question how to construct a suitable proxy distribution Tg.
Prioritized Proximal Experience Regularization. When the proxy distribution is uniform, this method approximates the maximum entropy consistency policy (MaxEnt CP). It should noted that the difference between the proxy distribution and the optimal policy distribution in optimizing the above objectives. When the proxy distribution is far from the optimal policy or the samples converge
lead to difficulty
proxy distribution is complex, the above optimization objectives require better results. To better balance training stability and sample efficiency, we propose the prioritized proximal experience regularization (PPER). Specifically, when sampling data from the replay buffer, we design sampling weight /3 for each data instead of sampling the data uniformly.
(8)
1 exp (2a a
B At
where is the hyperparameter and At is the interval between the sample generation step and the current step. [B| is the capacity of the replay buffer. The curves with different a are shown in Figure (b). In the above settings, data closer to the current step will be sampled with higher probability, while data farther away will have lower probability. We refer the above sampling method as prioritized proximal experience (PPE).
For policy evaluation, we suggest a distributional value function instead of deterministic value function to ensure the stability and accuracy of the value estimation. Precisely, we follow (56) and use mixture of Gaussian (MoG) to model the distribution of the state-action value. When MoG employed for value distribution, the following loss is used to update Q-function.
M log ((0
2/ 2//0 ((0111
(d) (E. 1,
( (re(se, at) + + where
M
a1+1 ~ To (8(+))))
1=1
where 7(e1,a)) is the estimated value distribution. According to the equation 9,, we need to sample M target Q-values 2/ and update the value distribution. Different from [56]] we sample only one next action a++1 instead of multiple actions to reduce the time cost and find that this simplification can achieve good experiment results. Considering the simplicity and efficiency of DrQ-v2 [57], our proposed consistency policy with prioritized proximal experience regularization (CP3ER) built based DrQ-v2. The framework is shown as Figure 3 (a). We sample the data from the replay buffer, and augment the image with the techniques DrQ-v2. Thanks to the natural randomness of the action from consistency policy, method no longer requires additional exploration strategics.

---

In addition, we only used single Q-network to estimate the mean and variance of the mixture of Gaussian instead of double Q-network. We consider prioritized proximal experience regularization when updating consistency policy and used equation (9) when training the Q-network, which differs from DrQ-v2. The complete algorithm is included in the appendix B.1
6 Experiments
In this section, we evaluate the proposed method from the following aspects: 1) Does CP3ER have performance advantages compared to current SOTA methods? 2) Can policy regularization improve the behavior of the policy? 3) What is the impact of different modules on the performance?
6.1 Visual Continuous Control Tasks
Environment Setup. We evaluate the methods on 21 visual control tasks from DeepMind control suite [58] and Meta-world [59]. We split these tasks into three domains, including medium-leve tasks in the DeepMind control suite, hard-level tasks in the DeepMind control suite, and 6 tasks in the Meta-world. The details of each domain are included in the appendix [
Baselines, We compare current advanced model-free visual RL. methods, including DrQ-v2 [57] ALIX 60) and TACO [6]] The more detailed results are shown in the appendix
6.1.1 Does CP3ER have performance advantages compared to current SOTA methods?
Medium-level tasks in Deep control suite. We evaluate CP3ER on 8 medium-level tasks (57 in Deep.indd control suite. The results are shown in Figure 4 From the left part of Figure it can be seen that compared the current SOTA methods, our proposed CP3ER has achieved better sample efficiency. It should be noted that TACO uses auxiliary losses of action and state representation during training to improve sample efficiency. Moreover, our proposed CP3ER uses no additional losses exploration strategies. On the right part of Figure 4 we compare the mean, Interquartile Mean (IQM), median, and optimal gap of these methods. CP3ER has significant advantages in all metrics and has more minor variance. This means that CP3ER has better training stability.
0r0+22
DrQ
AUX
Normalized Score
0.8
CPJER:
0.6
0.7
0.8
0.72
0.80
0.88
Normalized Score
0.4
Madian
Optimal_gap
DrQ-2
TACO
0.0
CP3ER
0.0
0.5
1.0
1.5
2.0
0.7
0.8
0.9
0.1
0.2
0.3
Number Steps millions)
Normalized Score
Figure Results on medium-leved tasks in DeepMind control suite with random seeds.
Hard-level tasks in DeepMind control suite. We also evaluate CP3ER on 7 challenging tasks 57 in the DeepMind control suite. should noted that all tasks here only train 5M frames, rather than the commonly used 30M frames in other work 53 [5]] This means it is a very hard challenge. From the results on the left part of Figure 5 it can be seen that most methods have yet to learn effective policy within 5M frames. Our proposed CP3ER surpasses the performance of all methods without relying on any additional loss or exploration strategies. Moreover, it has significant advantages on all metrics including mean, IQM, median, and optimal gap.
Meta-world, We also evaluated the methods on 6 complex tasks the Meta-world. The results are shown in Figure 6 We record the success rates of the tasks, and all results are based on the success rates. Compared to other methods, CP3ER can quickly learn effective manipulation policy, and the success rate can reach nearly 100% in all tasks within 2M steps. By comparing the mean, IQM, median, and optimal gap of the success rates, CP3ER has a significant performance advantage with minimal variance.

---

0.8
Blowd Tioo
DrQ-22
Mean
10M
ALIX
IQM Normalized memalleed Store
0.6
CPJER
0.75
Normalized Score
0.4
Median
Optimalgap
0.2
DrQ-v21
AUX
0.0
CPЗER
4
0.50
Number of Steps (in millions)
Normalized Score
Figure 5: Results on hard-level tasks in DeepMind control suite with random seeds.
1.0
Aue
DrQ-v2
Mean Normalizd Score
0.8
TACO
CP3ER
0.25
0.50
0.75
1.00
0.6
0.9
Normalized Score
0.4
Median
Optimal_app
DrQ-22
0.0
TACO
0.3
Steps millions)
1.5
0.3
0.6
Normalized Score
0.9
0.25
0.50
0.75
Figure 6: Results on Meta-world tasks with 5 random seeds.
6.2
Ablation Study
6.2.1 Can policy regularization improve the behavior of the policy during training?
Action distribution analysis with toy example. In order to further explore the impact of policy regularization on the training, we borrow the ID continuous bandit problem (9] to analyze the policy behavior. The green curve in Figure (a) shows the reward function. Within narrow range of actions, the agent receives higher rewards, while within a broader range, the agent can only receive suboptimal rewards. Therefore, the policy needs strong exploration ability to achieve the highest return. We compare Gaussian policy with entropy regularization (MaxEnt GP), Consistency-AC 33 and consistency policy with entropy regularization (MaxEnt CP), and record the action distribution during the training. As shown in Figure Co quickly converges to the local optimal value with the Q-loss. Policy regularization ensures the diversity of action distribution during consistency policy training, preventing the policy from falling into local optima too early. Moreover, consistency policy has robust exploration compared to the Gaussian policy and achieves higher returns.
Medent GP
1.0
Comistinay AC
0.8
0.6
Maditt
0.2
MaxEng GP
MaxEnt Consistency-AC
1000 2000 3000 4000 5000 6000 Number of Steos
(a) Action distribution of the different policies
(b) Returns of the policies
Figure 7: Results on the toy example. Left part is action distributions during training, while right returns of different policies.

---

Dormant ratio analysis. We have shown that the Q-loss can rapidly increase the dormant ratio of the consistency policy network, leading to a loss of policy diversity. In order to analyze whether entropy regularization can alleviate the phenomenon, we record the dormant ratios of the policy networks during the training the dormant rate in Consistency-AC, CP3ER has lower dormant ratio, which means that entropy regularization can effectively reduce the dormant ratios of consistency policy.
tasks. The results are shown in Figure Compared to the rapid increase in
CHIRLC D4t
Dorment Radio
Average Return
Dormant Ratio
Aveer Return
Dormant Ratto
Aver Returm
Gradient
Gradient
60..0
30k
Gradient Steps
(a) Acrobot-swingup
(b) Reacher-hard
(c) Dog-stand
Figure 8: Dormant ratios of the policy networks on different tasks with 5 random seeds.
6.2.2 What is the impact of different modules on the performance?
We conduct ablation studies in 2 tasks to evaluate the contribution of each module in the proposed method. In addition, to analyze the impact of proxy distribution for policy regularization on per- formance, also compare several candidates, including uniform distribution or behavioral policy in the replay buffer. The results are shown in Figure 9 It is noticeable that policy regularization crucial for consistency policy. Without policy regularization, consistency policy (CP3ER w/o PPER) makes difficult to learn meaningful behavior the tasks. The proxy distribution also has an impact on the performance. Using uniform distribution to regularize policies can make the policy (CP3ER w. MaxEnt) improvement difficult, resulting in low sample efficiency. Compared to using behavior distribution in the replay buffer (CP3ER URB), the policy (CP3ER) obtained through prioritized proximal experience sampling has closer distribution to the current policy, making policy optimization easier and resulting in higher sample efficiency. In addition, we find that the performance of CP3ER is significantly better than the baseline (DrQ-v2), indicating that the feasible usage of consistency policy can help solve visual RL tasks.
1000
500
800
Average Return
400
Average Return
600
300
400
200
OP3ER Maotin GPIER USB
200
PPER
OP3ER
0
0K
500K
1M
1M
2M
0K
500K
1M
1M
2M
TotalEnvinteracts
TotalEnninteracts
(a) Acrobot-swingup
(b) Reacher-hard
Figure Results of ablation study on visual control tasks with random seeds.
7 Conclusion
In this paper, we analyze the problems faced by extending consistency policy to visual RL under the actor-critic framework and discover the phenomenon of the collapse of consistency policy during training under the actor-critic framework by analyzing the dormant ratio of the neural networks. To address this issue, propose consistency policy with prioritized proximal experience regularization (CP3ER) that effectively alleviates the training collapse problem consistency policy. The method evaluated on 21 visual control tasks and shows significantly better sample efficiency and performance

---

than the current SOTA methods. It is worth mentioning that, to the best of our knowledge, our proposed CP3ER the first method to apply diffusiinn/onsiitencyy models visual RL tasks.
Our experimental results show that the consistency policy benefits from its expressive ability and ease of sampling, effectively balancing exploration and exploitation in RI. with high-dimensional state space and continuous action space. It achieves significant performance advantages without any auxiliary loss and additional exploration strategies. We believe that consistency policy will play an essential role in visual RL. There are still some issues worth exploring in future work. Firstly, auxiliary losses for representation in current visual RL have the potential improve the performance of consistency policy. Secondly, the diversity of behavior in consistency policy is crucial for RL exploration. This paper only discusses the stability of policy training and does not analyze the diversity of behavior during training, which will help improve the exploration performance of policies. In addition, consistency policy under the on-policy framework also a direction worth exploring.
Acknowledgments
This work supported by the National Natural Science Foundation of China (NSFC) under Grants No. 62103409, No. 62136008, No. 62293545 and in part by the International Partnership Program of the Chinese Academy of Sciences under Grant [04GJH22022013GC.
References
[1] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nat., 518(7540):529- 553-533,3233 2015.
[2] David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, loannis Antonoglou, Vedavyas Panneershelvam, Mare Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lillicrap, Madeleine Leach, Koray Kavukeuoglu, Thore Graepel, and Demis Hassabis. Mas- tering the game of go with deep neural networks and tree search. Nat., 529(75877:484-489, 2016.
[3] David Silver, Thomas Hubert. Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 11204419)11440-11 2018. doi: 1112//cciencc.aar4404..
[4] Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre,, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy P. Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with a learned model. Nat., 588(7839):604 609, 2020.
[5] Haoran Li, Qichao Zhang, and Dongbin Zhao. Deep reinforcement learning-basee automatic exploration for navigation in unknown environment. JEEE Transactions on Neural Networks and Learning Systems, (6)):2064-2076 2020. doi: 10. 109//N99NSSS... N02TTNNNSS.009...9 2019..92789999
[6] Chuanyu Yang. Kai Yuan, Qiuguo Zhu, Wanming Yu, and Zhibin Li. Multi-expert learning of adaptive legged locomotion. Sei. Robotics, 5(49):2174, 2020.
[7] Peter R. Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subramanian, Thomas Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, Leilani Gilpin, Piyush Khandelwal, Varun Kompella. HaoChih Lin, Patrick MacAlpine, Declan Oller, Takuma Seno, Craig Sherstan, Michael D. Thomure, Houmehr Aghabozorgi, Leon Barrett, Rory Douglas, Dion Whitehead, Peter Dürr, Peter Stone, Michael Spranger, and Hiroaki Kitano. Outracing champion gran turismo drivers with deep reinforcement learning. Nat., 602(7896): 223-228, 2022.
10

---

[8] Guangzheng Hu, Haoran Li, Shasha Liu, Yuanheng Zhu, and Dongbin Zhao. Neuronsmae: A novel multi-agent reinforcement learning environment for cooperative and competitive multi- robot tasks. In 2023 International Joint Conference on Neural Networks (IJCNN), pages 1-8, 2023. doi: 10. 109/UCNN555540002222.
[9] Zhiao Huang, Litian Liang. Zhan Ling, Xuanlin Li, Chuang Gan, and Hao Su. Reparameterized policy learning for multimodal trajectory optimization. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, PMLR, 2023.
volume 202 of Proceedings of Machine Learning Research, pages
[10] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 1352-1361. PMLR, 2017.
[11] Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tier Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information P Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[12] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 10674-10685. IEEE, 2022.
[13] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with CLIP latents. CoRR, abs/2204.00 06125,61155500 12,112222222 2022.
[14] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
[15] Jonathan Ho, Tim Salimans, Alexey A. Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances Neural Information Processing Systems 35: Annual Conference on Neural Information P Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
[16] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Do- minik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable video diffusion: Scaling latent video diffusion models to large datasets. CoRR, abs/2311.15127,, 2023.
[17] Zhengbang Zhu, Hanye Zhao, Haoran He, Yichao Zhong, Shenyu Zhang, Yong Yu, and Weinan Zhang. Diffusion models for reinforcement learning: A survey. CoRR, abs/2311 abs/2311.01122, 101223,,, 2023.
[18] Tim Pearce, Tabish Rashid, Anssi Kanervisto, David Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, and Sam Devlin. Imitating human behaviour with diffusion models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[19] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Kostas E. Bekris, Kris Hauser, Sylvia I. Herbert, and Jingjin Yu, editors, Roboties: Science and Systems XIX, Daegu Republic of Korea, July 10-14, 2023, 2023.
11

---

[20] Michael Janner, Yilun Du, Joshua B. Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learn- ing, 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 99159 PMLR, 2022.
[21] Zhixuan Liang, Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, and Ping Luo. Adaptdif- fuser: Diffusion models adaptive self-evolving planners. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardd,, Sivan Sabato, and Jonathan Scarlett, editors, Interna- tional Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 20725-20744 PMLR, 2023.
[22] Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. CoRR, abs/2310.061 14, 2023.
[23] João Carvalho, An T. Le, Mark Baierl, Dorothea Koert, and Jan Peters. Motion planning diffusion: Learning and planning of robot motions with diffusion models. In IROS, pages 1916-1923, 2023.
[24] Wonjoon Goo and Scott Niekum. Know your boundaries: The necessity of explicit behavioral cloning in offline RL. CoRR, abs/2206.00695, 2022.
[25] Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, OpenReview.neh, 2023,
[26] Philippe
Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. IDQL: implicit q-learning actor-critic method with diffusion policies. CoRR, abs/2304.10573, 2023.
[27] Cong Lu, Philip J. Ball, Yee Whye Teh, and Jack Parker-Holder. Synthetic experience replay. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023.
[28] Haoran He, Chenjia Bai, Kang Xu, Zhuoran Yang, Weinan Zhang, Dong Wang, Bin Zhao, and Xuelong Li. Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
[29] Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. The dormant neuron phenomenon in deep reinforcement learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages
PMLR, 2023.
[30] Michael Psenka, Alejandro Escontrela, Pieter Abbeel, and Yi Ma. Learning diffusion model policy from rewards via q-score matching. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceed- ings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 41 161663444 41 182. PMLR, 21-27 Jul 2024.
[31] Zhendong Wang, Jonathan J. Hunt, and Mingyuan Zhou. Diffusion policies expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net 2023.
[32] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 32211-32255 PMIRR 23-29 Jul 2023.
12

---

[33]
Zihan Ding and Chi Jin. Consistency models as a rich and efficient policy class for reinforcement learning. In The Tweifth International Conference on Learning Representations, 2024.
[34] Yuhui Chen, Haoran Li, and Dongbin Zhao. Boosting continuous control with consistency policy, In Mehdi Dastani, Jaime Simão Sichman, Natasha Alechina, and Virginia Dignum, editors, Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2024, Auckland, New Zealand, May 6-10, 2024, pages 335-344, ACM, 2024.
[35] Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. IDQL: implicit q-learning actor-critic method with diffusion policies. CoRR, 1ls/2304.10573, 2023.
[36] Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023.
[37] Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact energy- gerggygguided diffusion sampling in offiie reinforcement learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barhara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning. ICMI. 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 22825-22855 PMLR, 2023.
[38] Zhendong Wang, Jonathan J. Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[39] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B. Tenenbaum, Tommi S. Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[40] Haoran Li, Yaocheng Zhang, Haowei Wen, Yuanheng Zhu, and Dongbin Zhao. Stabilizing diffusion model for robotic control with dynamic programming and transition feasibility. IEEE Transactions on Artificial Intelligence, 5(9):4855 4594, 2024. doi: 10. 10.1109/A1.202443884000
[41] Long Yang, Zhixiong Huang, Fenghao Lei, Yueun Zhong, Yiming Yang, Cong Fang, Shiting Wen, Binbin Zhou, and Zhouchen Lin. Policy representation via diffusion probability model for reinforcement learning. CoRR, 22tp22355.11222 2023.
[42] Mare Rigter, Jun Yamada, and Ingmar Posner. World models via policy-guided trajectory diffusion. Transactions on Machine Leurning Reseurch, 2024. ISSN 2835-8856.
[43] Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 621 2020, virtual, 2020.
[44] Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regular- izing deep reinforcement learning from pixels. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[45] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continu- ous control: Improved data-augmented reinforcement lcarning. In The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. Open- Review.net, 2022.
[46] Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: contrastive unsupervised repre- sentations for reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, ICML. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 1639-5650, PMLR, 2020.
13

---

[47] Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR,
2018.
[48] Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 9870-9879. PMLR, 2021.
[49] Ruijie Zheng, Xiyao Wang, Yanchao Sun, Shuang Ma, Jieyu Zhao, Huazhe Xu, Hal Daumé III, and Furong Huang. TACO: temporal latent action-driven contrastive loss for visual reinforce- ment learning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023.
[50] Max Schwarzer, Ankesh Anand, Rishab Goel, R. Devon Hjelm, Aaron Courville, and Philip Bachman. Data-efficient reinforcement learning with self- predictive representations. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May
3-7, 2021. OpenReview.net, 2021.
[51] Edoardo Cetin, Philip J. Ball, Stephen J. Roberts, and Oya Çeliktutan. Stabilizing off-policy deep reinforeement learning from pixels. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 2784-2810. PMLR, 2022.
[52] Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Mare G. Bellemare, and Aaron C. Courville. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[53] Guowei Xu, Ruijie Zheng, Yongyuan Liang, Xiyao Wang, Zhecheng Yuan, Tianying Ji, Yu Luo, Xiaoyu Liu, Jiaxin Yuan, Pu Hua, Shuzhen Li, Yanjie Ze. Hal Daumé III, Furong Huang, and Huazhe Xu. Drm: Mastering visual reinforcement learning through dormant ratio minimization. In The Twelfth International Conference on Learning Representations, 2024.
[54] Johan Obando-Ceron Aaron C. Courville, and Pablo Samuel Castro. In value-based deep reinforcement learning, a pruned network is a good network. In International Conference on Machine Learning, 2024.
[55] Emlyn Williams and Athanasios Polydoros. Pretrained visual representations in reinforcement learning. CoRR, abs/2407.17233 2024.
[56] Bobak Shahriari, Abbas Abdolmaleki, Arunkumar Byravan, Abe Friesen, Siqi Liu, Jost Tobias Springenberg, Nicolas Heess, Matt Hoffman, and Martin A. Riedmiller. Revisiting gaus- sian mixture critics in off-policy reinforcement learning: sample- based approach. CoRR, abs/2204. 10256, 2022.
[57] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continu- ous control: Improved data-augmented reinforcement learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. Open- Review.net, 2022.
[58] Yuval Tassa, Yotam Doron. Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancy. Timothy P. Lillicrap, and Martin A. Riedmiller. Deepmind control suite. CoRR, abs/1800 11111111.0000 .00690, 2018.
[59] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura, editors, 3rd Annual Conference on Robot Leurning, CoRL. 2019, Osaka, Japan, October 30 November 1, 2019, Proceedings, volume 100 of Proceedings of Machine Learning Research, pages 1004-1100. PMLR, 2019.
14

---

[60] Edoardo Cetin, Philip J. Ball, Stephen J. Roberts, and Oya Çeliktutan. Stabilizing off- off-policy deep reinforeement learning from pixels. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song. Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, 1C21,2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 2784-2810. PMLR, 2022.
[61] Ruijie Zheng, Xiyao Wang, Yanchao Sun, Shuang Ma, Jieyu Zhao, Huazhe Xu, Hal Daumé III, and Furong Huang. TACO: temporal latent action-driven contrastive loss for visual reinforce- ment learning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 16, 2023, 2023.
15

---

A More Results on Dormant Ratios
To demonstrate the phenomenon policy degradation Consistency-AC, we conduct experiments with different settings more tasks and analyze the effect on the dormant ratio. For analyzing the impact of non-stationary data distribution on consistency model training in online RL, we employ classic tasks from D4RL. The results are shown in Figure [0 It can be that the non-stationary distribution caused by online training does not significantly affect the dormant ratio of the policy network.
Dormant Ratio 5.4
Dormant Ratio a
Dormant
60
Gradient
Gradient
Gradient Steps
(a) Halfcheetah
(b) Walker2d
(c) Hopper
Figure 10: The dormant ratios of the policy under the online and offline training. All results are averaged over 4 random seeds, and the shaded region stands for standard deviation across different random seeds.
For analyzing the impact of the loss function and figh-ddmensional state input on the consistency policy, we conduct experiments on 5 tasks separately. Among them, the results of tasks are presented the main part, and the results of the remaining tasks shown here.
Figure shows the impact of different loss functions on the performance and the dormant ratio of consistency policy. Compared SAC policy, consistency policy with Q-loss achieves the higher dormant ratio and the worse performance. Especially on the Finger-turn-hard task, the policy under the Consistency-AC framework hardly learn any meaningful behavior, and the dormant ratios remain at a high value. Figure 12 shows the impact of different observation inputs on Consistency-AC.. Compared to state-based settings, image-based policy is more difficult learn meaningful behavior, and its dormant ratio also maintains a higher value.
senrdie DE
M50
0,2085
Dormant Retio
Averaage Return
Dome nt Rato
Dormant
P-
age
age
2
0:
c
0.3000
64
JE300 $320 ac030 20000
20022
30500
20033
Steps
Steps
(a) Cheetah-run
(b) Finger-turn turdh
(c) Walker-walk
Figure 1I: The dormant ratios of the policy with different training loss. All results are averaged over 4 random seeds, and the shaded region stands for standard deviation across different random seeds.
B Implementation Details
In this paper, we propose Consistency Policy with Prioritized Proximal Experience Regularization (CP3ER). which is built on the basis of DrQ-v2 and its framework is shown in Figure [3] Compared to DrQ-v2, its difference lies the use of Prioritized Proximal Experience (PPE) when sampling data from the replay buffer, and the of consistency policy instead of Gaussian policy in the
16

---

Jite E4
420
D3
1000
Mta
Average Return
Dormant Ratio
Average Return
Dormant Ratio
Dormant 04
04
15
24
Gradient Steps
Gradient
Gradient Steps
(a) Acrobot-swingup
(b) Reacher-hard
(c) Finger-turn-han
Figure 12: The dormant ratios of the policy with different observations. All results are averaged over 4 random seeds, and the shaded region stands for standard deviation across different random seeds.
actor. In addition, it employs mixture of Gaussians to model the value distribution rather than the deterministic double Q-networks in DrQ-v2.
0
at
Environment
Replay
Augmentation
Double Q- networks
Gaussian Policy
DrQ-v2
(a) DrQ-v2
0
Environment
Critic Gaussian
Actor
Replay Buffer
Augmentation
Consistency Policy
(o.
CP3ER
(b) CP3ER
Figure 13: Comparison between DrQ-v2 and CP3ER.
B.1 Procedure of the proposed algorithm
We have demonstrated the complete procedure of CP3ER in Algorithm 1 and
B.2 Hyperparameters
present summary of all the hyperparameters for CP3ER in Table where DMC is the abbre-
viation of DeepMind control suite. It is worth noting that for tasks in different domains, only the
17

---

Algorithm Algorithm of CP3ER
Input:
fe. no. Qs: parametric networks for encoder, policy and Q-function respectively.
aug: random shifts image augmentation.
T, B,
a. т: training steps, mini-batch size, learning rate, target update rate. Training routine:
for each timestep 1..T do
a, s- To (o(((()))
oc+1 P(- 0t)
B B (or, dee R(or, at), 0c+1)
(Or, (e. Tr, 01+1) B with PPE sampling method UPDATECRITIC(O:; dt, 0t+1)
UPDATEACTOR(0),0) aq)
end for
Algorithm 2 Training critic
procedure UPDATECRITICCO:, ae, 0c+1)
h4s h441 + fe fe(aug(o))), (aug(or41))
a141 + (0(4))1)
Compute (6) using Equation 9
t a Lg
t Ø a (
- (1 T + TO
end procedure
learning rate and feature dimension are different, while other parameters are the same for all tasks. The parameters of our proposed method are not task-sensitive, which helps it be applied to wider range of visual control tasks without the need for fine-tuning of parameters.
C More Results
In this section, we present more detailed experimental results on tasks in DeepMind control suite and Meta-world, including performance curves during the training, performance profiles, and probability of performance improvement. We compare CP3ER to 4 baselines including DrQ-v2 (57), ALIX (60).
TACO [T and DrM [53] All evaluations are based on a single NVIDIA GeForce RTX 2080 Ti. For CP3ER, training run with 2M frames on this device will take about 13 hours.
DrQ-v2:
ALIX: http::///ttttcooool///o////a/aaa
TACO:
. DrM:
http://ggttabb.cco/usikkkk/////
Algorithm 3 Training actor
procedure UPDATEACTOR(O,.g) a.)
f; < (o:)
à 5 TT (o.)
Compute C7 using Equation
0 - 0 a (OL)
0 78
end procedure
18

---

Parameter
Setting
Replay buffer capacity
106
Action repeat
2
Seed frames
4000
Exploration steps
10000
n-Step returns
3
Mini-batch size
256
Discount ?
0.99
Optimizer
Adam
Learning rate
8 x 10 (Hard-level tasks in DMC)
10 (Medium-level tasks in DMC & Meta-world)
Soft update rate Features dimension
0.01
100 (Hard-level tasks in DMC)
50 (Medium-level tasks in DMC & Meta-world)
Hidden dimension
1024
Number of Gaussian mixtures for Critic
3
Number of samples M for update Critic
20
Parameter a for PPE
2.0
Coefficient for consistency loss 1)
0.05
T-Dormant ratio
0.025
Table I: The hyper-parameters for CP3ER.
C.1 Results on Medium-level Tasks in Deep.Mind Control Suite
In this subsection, show the detail results on the 8 medium-level tasks [57] in DeepMind control suite. These tasks include: acrobot-swingup. cheetah-run, finger-turn-hard, hopper-hop, quadruped- run, quadruped-wall reacher-hard and walker-walk. From the results in Figure [4] it be seen that our proposed CP3ER has better sample efficiency and performance on almost all tasks, even though it does not use any loss for representation learning. addition the performance curve during the training, we also demonstrate the performance profiles at different checkpoints and the probability of performance improvement. According the results in Figure CP3ER is significantly superior to other baseline methods.
1002
TAC2
sco
520-
630
1100
23
000
1.0
22
03
20
67
25
23
(a) Acrobot-swingup
(b) Chectah-run
(c) Finger-turn-han
(d) Hopper-hop
1000
500
Aixt
800
s0c
200
(0) Quadruped-run
(f) Quadruped-walk
(g) Reachhr-bard
(h) Walker-run
Figure 14: Performance of CP3ER against baseline algorithms DrQ-v2, ALIX, and TACO on the
medium-level tasks in DeepMind control suite. All results are averaged over random seeds, and the shaded region stands for standard deviation across different random seeds.
19

---

Fradtin of soore >1
200k
500K
1m
1.00
Algorithm
Algenthm
DPIER.
TACO
OPEER
p-0.2
0.25
Br
AUX
D/Q
0.00
MAD
pr042
0.00
0.75 Normalized Score (1)
0.25
0.50
1.00 0.00
11.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Normalized Score (Y)
Normalized Score (v)
POC
(a) Performance profiles
(b) Probability of improvement
Figure 15: Performance profiles and probabilities of improvement of different methods.
C.2 Results on Hard-level Tasks in Deep.Mind Control Suite
DeepMind control In this subsection, we show the detail results on the 7 hard-level tasks 53
57
suite. These tasks include: dog-run, dog-stand, dog-trot, dog-walk, humanoid-run, humanoid-stand and humanoid-wal -aak. In addition to the comparison baselines considered in the previous subsection, we also consider DrM which achieves SOTA performance on hard-level tasks in DeepMind control suite by weight perturbation and exploration strategies. [ the results CP3ER against several baselines. It is difficult for DrQ-v2, ALIX, and TACO to learn meaningful policy within the 5M framework on these tasks. CP3ER achieves comparable performance to DrM on tasks: dog-run, dog-stand dog- trot, and dog- walk without any exploration strategy. In addition, CP3ER significantly outperforms DrM in another 3 more challenging tasks. From the results in Figure [6 CP3ER also significantly outperforms other basclines on the tasks. Compared to the SOTA method DrM for difficult tasks, CP3ER also has a performance improvement probability of over 70%.
DrO
200
(a) Dog-run
(b)
(c) Dog-trot
(d) Dog-war walk
783
630
SM
450
(e) Humanoid-run
(f) Humanoid-stand
gg) Humanoid-walk
Figure 16: Performance of CP3ER against baseline algorithms DrQ-v2, ALIX, TACO and DrM on hard-level tasks in DeepMind control suite. All results are averaged over random seeds, and the shaded region stands for standard deviation across different random seeds.
C.3 Results on Tasks in Meta-world
In this subsection, we show the detail results on the hard tasks [53 in Metat-worid, These tasks include: assembly, disassemble, hammer, hand insert, pick place wall and stick pull. In these tasks, we compared 4 baselines, including DrM. It should be noted that in (53), DrM achieves SOTA performance on these hard tasks in Meta-world, but we cannot reproduce the corresponding performance based on the official codes. Therefore, this result is used for reference and may cannot represent the true performance of DrM. According to the results in Figure 18 and Figure 19 CP3ER achieves SOTA performance on hard-level tasks Meta-world, and compared the baselines, CP3ER has significant performance advantage.
20

---

Fradtin of soore >1
4m
1.00
ALK
TACO
Algorithm
CR3ER
Algarithm
1A00
07201
CR3E5
OPE6
DmN
Dro+2
1863
Dro+2
0.00
D10
Dmh
0.00
0.25
0.50
0.75
1.00 0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Normalized Score (X)
Normalized Score (Y)
Normalized Score (v)
(a) Performance profiles
(b) Probability improvement
Figure 17: Performance profiee and probabilities of improvement of different methods.
1.0
1.0
1.0
TAC0
Moan Sucoossfall
Ween Sucresstas
CPSER
010
0.6
0.6
AIX
0.5
ALE
TUCO
TAYO
0.4
Mean
RMM
D-N
(PTTR
0.2
0.2
0.2
0.0
0.0
16
DraiEriinterads
TotalEnvinteraccs
(a) Assembly
(b) Disassemble
(c) Hammer
1.0
1.0
1.0
DrQ-
Succussuute
0.8
SuccessRat
TACO
Mean Successtate
DYH
CPSER
CPIER
0.6
0.6
0.5
0.4
dro42 Aux
0.2
TACo
0.2
0.2
0.0
0.0
0.0
Trtanivineeaacss
14
TntalEniinteraads
TotstEnvinteract
(d) Hand insert
(e) Pick place wall
(f) Stick pull
Figure 18: Performance of CP3ER against baseline algorithms DrQ-v2, ALIX, TACO and DrM on tasks in Meta-world. All results are averaged over 5 random seeds, and the shaded region stands for standard deviation across different random seeds.
>1
500k
steps
1.00
Algorithm
TATO
CFFEA
(ACO
0.50
(F363
DnN
0.25
AlE:
bro
TACO
Dro
0.00
D10
DYN
0.00
0.25
0.50
0.75
1:00
0.00
0.25
0.50
0.75
1000
0.00
0.25
0.50
0.75 1.00
Normalized Score (1)
Normalized Score (r)
Normalized Score (t)
(a) Performance profiles
(b) Probability improvement
Figure 19: Performance profiles and probabilities of improvement of different methods.
C.4 Results on State-based Tasks
Sample efficiency has always been a highly concerned issue in visual RL, and high expressive power of the policy can improve exploration efficiency and thus increase sample efficiency. After introducing the diffusion model into visual RL, we find serious phenomenon of policy degradation, which results in disastrous performance. Therefore, we propose CP3ER to stabilize the training of consistent policy. In addition visual RI. tasks, CP3ER theoretically has the potential to be applied to state- based RL tasks. According the experimental setup in [34], conduct comparison to
21

---

state-based RL SOTA methods, and the experimental results are shown in the Table 2 It can be seen that compared the current SOTA methods. CP3ER also has significant advantages in the tasks, which also proves the generalization of CP3ER different observation tasks.
Tasks
TD3
SAC
PPO
MPO
DMPO
D4PG
DreamerV3
CPQL
CP3ER
Acrobot Swingup
46.8
33.2
34.4
80.6
98.5
125.5
154.5
183.1
362.9
Finger Turn Easy
337.6
371.4
275.2
430.4
593.8
524.5
745.4
874.1
867.45
Finger Turn Hard
334.4
344,8
5.06
250.8
384.5
379.2
841.0
864.6
912.65
Hopper Hop
40.0
41.7
0.0
37.5
71.5
67.5
111.0
130.1
299.3
Hopper Stand
322.7
270.9
2.2
279.3
519.5
755.4
573.2
902.1
904.1
Walker Run
274.5
445.9
131.7
539.5
462.9
593.1
632.7
683.8
646.5
Average
226
251.3
74.8
269.7
355.1
407.5
509.6
606.3
665.5
Table 2: Comparison CP3ER and other methods on state-based RL tasks in DeepMind control suite.
In addition, we also compare the existing methods based on diffusion/consistency models following the settings [33), and the results are shown in Table It can be seen that compared to existing methods based on diffussinn/consiseency models, our method CP3ER, although aimed at visual RL problems, still has significant advantages when extended state-based RL tasks.
Online tasks
Diffusion-QL
Consistency-AC
CP3ER
Halfcheetah-m
338..
6725.211944.4
10699.1 +: 1054.7
Hopper-m
3675.5
3589.7+1133.
3309.6 6 133.2
Walker2d-m
4316.2-612 612.1
3790.9+1677.5
5201.0+111.3
Average
4579.2
4701.9
6403.2
Table 3: Comparison of CP3ER with diffussion/consisteney based RL methods.
D Limitations
Although CP3ER enhances the training stability of consistency policy in the actor-critic framework and achieves excellent performance visual control tasks, the policy diversity of CP3ER has not been thoroughly explored. In CP3ER, this diversity only depends on the multimodal actions in the replay buffer, which gradually disappears as the training steps increases. Therefore, CP3ER may face the risk of losing diversity in consistent policy, thereby weakening its exploration ability certain extent. In addition, there is a lack of theoretical analysis on CP3ER. Although it is based on the actor-critic framework, its policy improvement and convergence property require more rigorous theoretical analysis.
E
Broader Impacts
This work mainly focuses on the field of visual RL and proposes new method that may significantly improve the efficiency of visual RL. This method may improve the efficiency of robot skill learning and have a wide impact on visual control fields such as robots, but will not involve ethical and safety issues. Therefore, this work should not have negative social impacts.
22

---

NeurIPS Paper Checklist
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatiin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors their evaluation.
While "[Yes] " is generally preferable to "[No] it perfectly acceptable to answer "[No] provided proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our claims stated in the abstract and introduction are consistent with the contributions and scope of our paper as detailed in Section !
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper acknowledges its limitations in appendix D
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
23

---

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specinf well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tes system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
There are no assumptions or proofs presented in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper fully discloses all necessary details for reproducing the main experimental results, including comprehensive descriptions of the algorithm in section 5nnd and appendix B.I. experiments setup in section G and parameters used in appendix B.2
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.

---

If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
We have submitted the code for training as the supplementary material, and once the paper is accepted, the code will be fully open source.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
pwt.huooommisiiooooocoocc for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
25

---

Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes)
Justification: The paper provides specific details about hyperparameters settings in appendix
B.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper comprehensively presents statistical significance in the results in sections andd appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper specifies the type of compute workers (a single NVIDIA GeForce RTX 2080 Ti) in appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
26

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics sttts://neur/i:
Answer: [Yes]
Justification: We have reviewed the NeurlPS Code of Ethics. We confirm that the work involved in the paper complies with Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The paper acknowledges its broader impacts in appendix D
Guidelines:
The answer NA means that there no socictal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our research is mainly focused visual RI., with the main potential ap- plication direction being robot skill learning. Therefore, this paper should not have the aforementioned risks.
Guidelines:
27

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
We have cited relevant papers and provided explanations for the referenced code in our codes. The majority of our code is licensed under the MIT license, however portions of the project available under separate license terms: DeepMind licensed under the Apache 2.0 license.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI..
The name of the license (e.g., CC-BY 4.0) should be included for each
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have included a README file the provided code, which provides detailed instructions on deployment, training, and license.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
28

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "UL40LCXKN3", "reviewer": "Reviewer_TpQn", "summary": "The paper analyzes the problems faced by extending consistency policy to visual RL under the actor-critic framework and discovers the phenomenon of the collapse of consistency policy during training under the actor-critic framework by analyzing the dormant rate of the neural networks. The authors propose a consistency policy with prioritized proximal experience regularization (CP3ER), which employs entropy regularization to constrain policy behavior. The experiments evaluate CP3ER with recent baselines (DrQ-v2, ALIX, TACO), on 21 visual control tasks.", "questions": "- According to [1], dormant rates in reinforcement learning and supervised learning represent different meanings. Can you explain in detail how dormant rates affect the expression ability of the consistency policy during training? For example, what does a high dormant rate indicate? What does an increasing dormant rate signify? And why?\n    - [1] Sokar, Ghada, et al. \"The dormant neuron phenomenon in deep reinforcement learning.\" *International Conference on Machine Learning*. PMLR, 2023.\n- In Sec. 4, Fig. 3, since Q-loss under the actor-critic framework will destabilize the consistency policy training, why was only the Q-loss used during the training process?\n- How do the computational cost and runtime compare with other baselines?", "limitations": "See the weaknesses and questions section.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and organized, and includes a thorough discussion of the relevant related works.\n- Sec. 4 analyzes the Consistency Actor-Critic from the perspective of dormant rates, which is very interesting.\n- The experimental results showcase impressive performance gains over the state-of-the-art on a wide range of robotics tasks (e.g., DM-Control, Meta-World). Ablations are also provided to delineate the impact of each modification.", "weaknesses": "- The method is not directly applicable to domains with discrete action spaces.\n- The experiments do not compare with other consistency RL methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and valuable comments. Please note the global response with the attached PDF. \n\n> The method is not directly applicable to domains with discrete action spaces.  \n\nIndeed, our method has not yet been extended to discrete action spaces because we use consistency models to model the policy, and most diffusion/consistency models are currently applied in continuous state spaces. Therefore, our proposed method is also subject to this limitation. However, to extend it to discrete action spaces, one possible approach could be to binarize the discrete actions and then map them to continuous spaces, as suggested in [1]. This could be an interesting and worthwhile direction for future exploration.  \n\n> The experiments do not compare with other consistency RL methods.  \n\nOur proposed method aims to address the issue of sample efficiency in visual RL. To the best of our knowledge, as of the time of paper submission, CP3ER is the only one to apply diffusion/consistency models to online visual RL policy. This is why we have not compared it with other RL methods based on diffusion models or consistency models. In the realm of state-based RL, there are already some methods based on diffusion models or consistency models, and we have compared our method with them. The results are shown in Table 3 of the attached PDF. Following the settings in [2], we compared CP3ER with Consistency-AC and online Diffusion-QL. As can be seen from the results, although our method is designed to visual RL problems, it still demonstrates significant advantages when extended to state-based RL tasks compared to existing methods based on diffusion or consistency models.  \n\n> Can you explain in detail how dormant rates affect the expression ability of the consistency policy during training? For example, what does a high dormant rate indicate? What does an increasing dormant rate signify? And why?  \n\nThe dormant ratio of a neural network represents the proportion of neurons that are inactive. The higher dormant ratio indicates fewer active neurons in the neural network, implying the network's capacity and expressiveness are damaged. The policy network's dormant ratio in RL is related to episode return. A higher dormant ratio hints at lazy action in RL with lower episode returns. Conversely, when the performance is good, the policy network is usually more active, and the dormant ratio is usually lower. This phenomenon has been observed in several studies [3-6].  \n \n> In Sec. 4, Fig. 3, since Q-loss under the actor-critic framework will destabilize the consistency policy training, why was only the Q-loss used during the training process?  \n\nIn the third part of Section 4, our main objective is to investigate whether consistency policy is suitable for visual RL tasks. Consistency-AC [2], a typical representative of consistency policy, uses only Q-loss during training. Therefore, we maintained the same settings in this part of the experiment. Fig. 3 illustrates the impact of different observations on the dormant ratio of the consistency policy network. The phenomena in the figure show that consistency policy in visual RL faces a very high dormant ratio, making policy training challenging. From Fig. 3, we can also conclude that Consistency-AC is not suitable for visual RL learning tasks.  \n\n> How do the computational cost and runtime compare with other baselines?  \n\nWe evaluated the training and inference time of CP3ER separately, comparing it with the methods mentioned in the paper. The GPU used is 2080Ti and the batch size for training is 256. The results are shown in the following table. Compared to TACO, our proposed CP3ER does not show a significant increase in training time and inference time costs. During training, the additional time cost of CP3ER compared to DrQv2 is mainly attributed to the computation of the regularization loss term.  \n\n\n|       | DrQv2 | ALIX | TACO | DrM | CP3ER |\n| ----------- | ----------- | ----------- | ----------- | ----------- | ----------- |\n|Training Time per Batch (s) | 0.0339 | 0.0332 | 0.1919 | 0.04075 | 0.04668 |\n| Inference Time per Step (s) | 0.00132 | 0.00141 | 0.00131 | 0.00184 | 0.00218 |  \n\n\n[1] Structured Denoising Diffusion Models in Discrete State-Spaces, ICLR 2023.  \n\n[2] Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning, ICLR 2024  \n\n[3] DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimization, ICLR 2024(Figure 2, 11)  \n\n[4] In deep reinforcement learning, a pruned network is a good network, Arxiv 2024(Figure 11)  \n\n[5] The Dormant Neuron Phenomenon in Deep Reinforcement Learning, ICML 2023 (Figure 9, 11, 16)  \n\n[6] Pretrained Visual Representations in  Reinforcement Learning, Arxiv 2024 (Figure3, 4)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UZ0vfrjLZ1", "reviewer": "Reviewer_Xhh1", "summary": "The paper addresses challenges in visual RL with high-dimensional state spaces, specifically focusing on improving sample efficiency and training stability. It introduces a novel method called CP3ER, which incorporates sample-based entropy regularization and prioritized proximal experience regularization to stabilize policy training and enhance sample efficiency. The proposed CP3ER achieves state-of-the-art performance in 21 tasks across the DeepMind control suite and Meta-world, demonstrating the effectiveness of applying consistency models to visual RL. The paper identifies and addresses the instability issues caused by the Q-loss in the actor-critic framework and the non-stationary distribution of online RL data. Experimental results show that CP3ER outperforms existing methods without relying on additional exploration strategies or auxiliary losses.", "questions": "Why using eq. (8) for sampling weight? Can other functions that shares a similar look demonstrated in Fig 4(b) also work?", "limitations": "Limitations have been discussed in the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The introduction of CP3ER, which combines consistency models with prioritized proximal experience regularization, presents a novel approach to addressing challenges in visual RL, enhancing both sample efficiency and training stability.\n2. The paper provides comprehensive empirical evidence demonstrating that CP3ER achieves state-of-the-art performance across a wide range of tasks in the DeepMind control suite and Meta-world.\n3. The paper provides a thorough analysis of the impact of non-stationary distributions and the actor-critic framework on consistency policy, offering insights into the underlying mechanisms that affect policy training stability.", "weaknesses": "In general, the paper is well-written and easy to follow. However, there is no illustration or theoretical support for using Eq. (8) for sampling weight. It would be better for the authors to consider more theoretical analysis on the solution design.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your recognition of our work.\n\n> Why using eq. (8) for sampling weight? Can other functions that shares a similar look demonstrated in Fig 4(b) also work?  \n\nIn fact, Equation (8) is an empirical formula inspired by the Sigmoid function, and it has been improved to meet the desired properties. We aim to sample the most recently collected data with a high probability during the sampling process while sampling older data with a relatively lower probability. If other formulas can satisfy the required properties (Fig. 4(b)), they may also be used to generate sampling weights.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1NKo0tu5RW", "reviewer": "Reviewer_UMZF", "summary": "This paper proposes a novel method that employs a consistency model for policy parameterization in online visual reinforcement learning settings. It initially identifies the issues of prior consistency actor-critic methods in high-dimensional observation online RL settings through the lens of dormant ratio. Subsequently, it proposes to incorporate entropy regularization alongside a prioritized sampling strategy to regularize the policy. Empirical experiments on 21 tasks across both the DeepMind Control Suite and MetaWorld demonstrate the effectiveness of the proposed method.", "questions": "- Although CP3ER is mainly designed for online visual RL, the proposed modifications (entropy regularization, prioritized replay, Gaussian mixture distribution of value functions) do not seem to be limited to visual observation settings. It would be interesting to see how the proposed algorithm performs in state-based RL settings.\n- For the 8 hard DeepMind Control Suite tasks and also Metaworld tasks, the authors compare CP3ER with TACO/ALIX/DrQ-v2 but not with DrM, which was reported in the original DrM paper to achieve good empirical performance on these tasks. This comparison here is missing.\n- For the analysis (Figures 2, 3, 9), it would be beneficial to include the episodic return in the plots so that readers can understand the numerical performance of the policy in addition to the policy network’s dormant ratio, which is an intrinsic factor not necessarily correlated with the policy’s episodic return.\n- The conclusion drawn from lines 202-203, “Therefore, we can infer that visual RL will exacerbate the instability of consistency policy training caused by the Q-loss under the actor-critic framework,” does not seem convincing. Even in ordinary policy", "limitations": "NA", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Overall, the paper is well-organized and presented. \n- The analysis of the limitations of existing consistency policy training in visual RL settings is thorough and convincing. \n- The experimental results across a wide range of tasks are promising compared to previous state-of-the-art methods.\n- The paper includes a good set of ablation studies to support the design choices of the proposed algorithm.", "weaknesses": "- Lack of baseline comparison in the experiments\n- The scope of the paper is limited to online and visual deep reinforcement learning\n- The proposed modification is straightforward compared with existing algorithms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and helpful suggestions. Please note the global response and the attached PDF.\n> The proposed modification is straightforward compared with existing algorithms.\n\nThe key challenge lies in addressing the instability during the training of consistency policy. By analyzing changes in the dormant ratio of the policy network, we observed a policy degradation phenomenon and speculate that this may be due to the instability of the data distribution and the score function represented by the Q-function. Therefore, we introduce entropy regularization. From the perspective of RL, our goal is to imbue the policy with specific attributes by modifying the loss, such as maximizing entropy to boost sample efficiency. From the perspective of matching distributions, entropy regularization provides a prior for the target data distribution, with the Q-function weighting this prior, thus stabilizing the optimization objective of the consistency model.\n> It would be interesting to see how the proposed algorithm performs in state-based RL settings.\n\nCP3ER can indeed be applied to state-based RL. Following [1], we selected 6 challenging tasks from the DeepMind Control Suite(DMC) environment to evaluate the methods. Table 1 in the attached PDF shows the results of different methods within 500k interacting steps. Compared to commonly used RL methods, CP3ER has significant performance advantages and also outperforms the current SOTA diffusion model-based method CPQL.\n\n> For the 8 hard DeepMind Control Suite tasks and also Metaworld tasks, the authors compare CP3ER with TACO/ALIX/DrQ-v2 but not with DrM, which was reported in the original DrM paper to achieve good empirical performance on these tasks. This comparison here is missing.\n\nDrM is indeed an excellent method for visual RL. In fact, we considered DrM in the appendix. Figures 14 and 15 show the comparison of DrM on DMC hard tasks. Figures 16 and 17 show the comparison results on Meta-world tasks. We did not include these results in the main part of the paper because when we reproduced DrM results using the official code[2], we could not achieve the performance claimed on Meta-world tasks. To obtain a fairer comparison, we compared our method and the reproduced results with the curves from [3], as shown in Table 2 of the attached PDF. Even compared to the results in the DrM paper, our method achieves comparable or even better performance. We have also included performance curves for several methods on DMC-hard and Meta-world tasks in the attached PDF(Fig. A, the left part shows curves on DMC-hard tasks and the right part shows curves on Meta-world tasks.). \n> For the analysis (Figures 2, 3, 9), it would be beneficial to include the episodic return in the plots so that readers can understand the numerical performance of the policy in addition to the policy network’s dormant ratio, which is an intrinsic factor not necessarily correlated with the policy’s episodic return.\n\nThank you very much for your suggestion. We have added an explanation of the relationship between the dormant ratio and policy performance after Section 3.3 in the revised version as shown in the global response.\n\nHowever, it is important to note that the dormant ratio is a measure of the capacity and the expressiveness of the network, and it is not a sufficient and necessary condition for assessing policy performance. A policy network with a low dormant ratio may have a low episode return. For example, when trained with supervised learning (such as consistency loss), the network typically exhibits a low dormant ratio, but since its goal is to fit the data distribution rather than to maximize average returns, its policy performance may not necessarily be superior. To adequately assess the policy degradation phenomenon caused by Q-loss, we compared the performance of the consistency policy trained with Q-loss to SAC (the online data collection policy required when training with consistency loss). We also compared the dormant ratios of the policy networks in SAC and redrew Figure 2 (Figure B middle and right, and Figure C in the attached PDF). Consistency policy with Q-loss has a higher dormant ratio compared to those trained with consistency loss and SAC.\n\nWe revised Figure 3 and 9 from the original paper with average returns as the other axis. The modifications are shown in Figures D and E in the attached PDF. The results from the figures indicate that generally, the higher the dormant ratio of the policy network, the poorer the performance of the policy. Visual RL tasks exacerbate the degradation phenomenon of consistency policies, resulting in poor policy performance.\n> The conclusion drawn from lines 202-203, “Therefore, we can infer that visual RL will exacerbate the instability of consistency policy training caused by the Q-loss under the actor-critic framework,” does not seem convincing. Even in ordinary policy\n\nIn the third part of Section 4, we analyzed consistency policy, particularly whether Consistency-AC is suitable for visual RL tasks. Figure 3 shows the dormant ratio of the policy network under different observations. Compared to state inputs, the network's dormant ratio rapidly rises to very high values with visual inputs, and the variance is small. This indicates that this phenomenon will stably exist, unaffected by random seeds.  Compared to state-based RL tasks, we can see that the policy degradation phenomenon of Consistency-AC in visual RL is quite severe. Consistency-AC adopts a Q-loss under the Actor-Critic framework. Thus, visual RL exacerbates the instability of consistency policies. In the modified revision, we add the explanation in 3.3 discussing the relationship between the dormant ratio and the policy performance.\n\n[1] Boosting Continuous Control with Consistency Policy, AAMAS 2024\n\n[2] https://github.com/XuGW-Kevin/DrM\n\n[3] DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimization, ICLR 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jMgCzPkol3", "reviewer": "Reviewer_ceLx", "summary": "The paper presents a novel method using consistency models as a policy parameterization to address online reinforcement learning. The paper first uses the dormant ratio metric to study properties of the consistency-AC method and determine that it is suitable for online RL. Then, it presents the CP3ER method, which adds an entropy regularization term to consistency AC in order to make it suitable for online RL. Finally the paper presents results showing strong performance in image-based RL benchmarks.", "questions": "Random Questions / Comments\n - I find the analysis related to figure 1 to be incomplete and hard to follow. \n    - First, it would be helpful to explain somewhere in your paper what we can infer about a neural network based on its dormant ratio, which would make this experiment easier to understand. Your paper seems to assume the reader is very familiar with Sokar et. al. but this need not be the case\n    - How big are the datasets? Where does the online dataset come from (ie what RL algorithm, how long was it trained, how often do you update the online dataset, etc)?\n    - Why is there a difference between online and offline dormant ratio for halfcheetah and not walker2d? And how can you draw conclusions from this experiment when you only run two environments that seem to get different results?\n - Figure 2 experiment is also missing a lot of details\n    - What AC algorithm does 'Q-loss' refer to?\n    - Where does the data for this experiment come from?\n    - Once again, I'm having trouble understanding how to interpret these plots or how you come to your conclusion.\n - Figure 3 has similar issues to the last two\n - The results as they are look nice but I have a few comments\n    - I think the set of baselines is incomplete. It would be nice to see a SOTA on-policy method like PPO\n    - All of the results show normalized scores, but you never explain how they are normalized, which makes them impossible to compare to other papers\n\nNit picks:\n - It would be nice if you could add one sentence in English describing what a dormant neuron is to section 3.3 for those who are not familiar with it. It would also be nice to explain why dormant ratio is important for the expressive capabilities of a network\n - Line 134: newtork", "limitations": "The authors address limitations and broader impacts of their work in the appendix.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Diffusion and consistency policies are an increasingly important research area, and in particular there has been limited work training these with online reinforcement learning. This paper seems to take an important step towards that direction, and thus has the potential to be significant.\n - Although several other works have used similar setups to apply DP/CP to offline RL, the specific formulation the authors employ to address online RL is novel as far as I can tell.\n - The method overall seems principled and the results presented in the paper seem promising.\n - Although there are issues with section 4 which I discuss extensively in the 'Questions' section, I like what the authors are trying to do and think if the section is written by more detail it can strengthen the case for their method.", "weaknesses": "- I think the biggest issue with the paper is that the story is a bit muddled. The core contribution of the paper seems to be CP3ER and the PPER regularization method. These things should be able to work for both low-dimensional and image-based observations. However, the authors specifically state that their goal is to address image-based observations. Why? Does this algorithm no longer show any benefit in these settings? The algorithm doesn't have any contributions that specifically focus on addressing image-based observations so I don't understand why that is a central focus of the evaluation.\n    - In order to properly contextualize this algorithm I think it would be necessary to compare it to low-dim baselines like SAC and PPO on a low-dim version of the benchmarks in addition to the state-based versions.\n - In the introduction the authors mention that generative models like diffusion and consistency are helpful for RL because they allow the policy to represent complex behaviors. However, none of the tasks in the training suite are 'complex' enough that they couldn't be solved by a simple MLP policy. It would be nice to see perhaps some tasks where a policy that can sample multimodal actions shines above MLP-based policies.\n - Section 4 is confusing and there are several other minor presentation issues that I explain in the 'Questions' section", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for the careful review and constructive feedback. Please note our global response with the attached PDF. In the modified revision, we expound on the correlation between the dormant ratio and the performance of the policy network in Section 3.3 as shown in the global response.\n> These things should be able to work for both low-dimensional and image-based observations. Does this algorithm no longer show any benefit in these settings? \n\nIn this work, we propose a proximal policy regularization method to stabilize the training of the consistency policy to improve the sample efficiency of visual RL. CP3ER indeed has the potential to be applied to state-based RL tasks. We supplement comparisons with SOTA methods[1]. The results are shown in Table 1 of the attached PDF. Compared to current SOTA methods, CP3ER demonstrates significant advantages in state-based RL tasks, further proving the generalization capability of CP3ER across different RL tasks.\n> It would be nice to see perhaps some tasks where a policy that can sample multimodal actions shines above MLP-based policies.\n\nDiffusion/consistency models build the policy distribution without modifying the structure of the policy network. Our policy network utilizes a simple MLP network, which is the same as the networks in Consistency-AC [2] and Diffusion-QL [3].\n> How big are the datasets? Where does the online dataset come from (ie what RL algorithm, how long was it trained, how often do you update the online dataset, etc)\n\nThe pre-collected dataset from D4RL (medium-replay-v2) is used as the offline data, which is from the replay buffer of SAC, comprising 1 million transition tuples. For online training, we employ SAC as the baseline and sample data from the replay buffer to train consistency policy with consistency loss. The replay buffer is updated with each interacting step. For Figure 2, we use the same setting for consistency loss.\n> Why is there a difference between online and offline dormant ratio for halfcheetah and not walker2d? And how can you draw conclusions from this experiment when you only run two environments that seem to get different results?\n\nThe dormant ratio change trends in the two subplots are the same, even though there are slight differences in the specific details. We speculate that the diversity of samples and actions in the halfcheetah dataset is lacking, leading the consistency policy to quickly fit the data distribution and overfit during offline training. In contrast, during the online training, due to the continuous updating of data, the policy network remains active. In the walker2d dataset, the diversity of the samples and action is high, requiring a certain training step to fit, resulting in the same as online learning.\n\nDue to dataset limitations (3 medium-replay datasets are available in D4RL), we conduct 3 tasks when analyzing the impact of offline/online settings on the dormant ratio of the policy. To analyze the effects of training losses and observations on the dormant ratio, we perform 5 different tasks, respectively. Curves not included in the main paper are available in the attached PDF, specifically in Fig. B and C.\n> What AC algorithm does 'Q-loss' refer to?\n\nQ-loss refers to Equation [2] in the paper, which is the loss used to update the policy.\n> (Figure 2 and 3) I'm having trouble understanding how to interpret these plots or how you come to your conclusion.\n\nWhen training the consistency policy using Q-loss, the variance in the dormant ratio of the policy network is quite large. This is because, under some random seed settings, the dormant ratio rapidly increases and remains at a relatively high level throughout. Subsequent training hardly changes the number of active neurons in the network, causing the policy to fall into a dormant where it outputs almost the same actions with different state inputs. With Fig. B and C in the attached PDF, this phenomenon indicates the performance descent. Compared to the state, the network's dormant ratio with the image quickly rises to high values with small variance, indicating severe degradation. With Fig. C in the attached PDF, we can observe that the policy degradation phenomenon of Consistency-AC in visual RL is severe. \n> It would be nice to see a SOTA on-policy method like PPO\n\nFor visual RL, sample efficiency is a common issue. As an on-policy method, PPO does not exhibit notable sample efficiency, which is why it has not been extensively studied in visual RL. Instead, there is more focus on off-policy methods, such as shown in [4-7]. Additionally, in state-based RL, PPO performs poorly compared to other methods(Table 1 in the attached PDF). Furthermore, our motivation is to improve sample efficiency in visual RL, we believe there is no need to compare with on-policy methods like PPO in visual RL tasks.\n> All of the results show normalized scores, but you never explain how they are normalized, which makes them impossible to compare to other papers.\n\nWe used the metrics recommended in [8] to analyze and compare the performance of algorithms.\nAdditionally, we provide the unnormalized IQM curves in the appendix (Figure 12, 14, and 16) for comparison with results from other papers.\n\n\n[1] Boosting Continuous Control with Consistency Policy, AAMAS 2024.\n\n[2] Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning, ICLR 2024\n\n[3] Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning, ICLR 2023\n\n[4] DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimization, ICLR 2024(Figure 2, 11)\n\n[5] Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning, ICLR 2021\n\n[6] TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learning, NeurIPS 2023\n\n[7] Stabilizing Off-Policy Deep Reinforcement Learning from Pixels, ICML 2023\n\n[8] Deep Reinforcement Learning at the Edge of the Statistical Precipice, NeurIPS 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**Please see the attached one-page PDF for additional experimental results.**

We would like to express our sincere gratitude for the efforts and valuable feedback provided by all the reviewers. We are very pleased that our work has been recognized by the reviewers, and we have also noted some points that have been commonly raised:

## Can CP3ER extend to state-based RL?

Both reviewers *UMZF* and *ceLx* have raised concerns regarding whether our proposed CP3ER can be generalized to state-based RL tasks. Our initial motivation was to address the issue of sample efficiency in visual RL. The high expressiveness of the policy helps to improve exploration and thereby enhances sample efficiency. Directly introducing diffusion/consistency models into visual RL results in severe policy degradation, preventing policy training. Therefore, we proposed a proximal policy regularization method to stabilize the training of consistency policies in visual RL. We also recognize that CP3ER indeed has the potential to be applied to state-based RL tasks. Consequently, we have included additional experiments in the attached PDF (Table 1 and Table 3) comparing CP3ER with SOTA methods in state-based RL. We found that CP3ER still exhibits advantages in state-based tasks. However, these advantages are not as pronounced as in visual RL tasks, because while policy degradation also occurs in state-based tasks using diffusion/consistency models, it is not as severe as in visual RL. These results also support the discussion in the last paragraph of Section 4 of our paper.

## What is the relationship between dormant ratio, policy degradation, and policy performance?

In Section 4, reviewers *TpQn*, *UMZF*, and *ceLx* care about how the conclusion of the policy degradation derived from the changes in the dormant ratio of the policy network. It must be acknowledged that in writing the paper, we neglected that readers might not be familiar with the concept of dormant ratio in RL. As introduced in [3], the dormant ratio of a neural network indicates the proportion of inactive neurons and is typically used to measure the activity of the network. A higher dormant ratio implies fewer active neurons in the network, implying the network's capacity and expressiveness are damaged. In RL, the episode return is closely related to the dormant ratio of the policy network. A higher dormant ratio results in more lazy action outputs, inactive agent behavior, and lower episode returns; conversely, when policy performance is good, the policy network is usually more active, and the dormant ratio is typically lower. This phenomenon has been reported in [3-6]. Based on these findings and the results in Section 4, Q-loss is identified as the primary reason for the increase in the dormant ratio of consistency policy. An increase in the dormant ratio often signifies a decline in policy performance (We have added additional curves of dormant ratio and performance in Fig. B, C, and D of the attached PDF. ), thus indicating policy degradation. According to the results in Figure 3 of the paper (Figure D in the attached PDF. Considering the page limit, the tasks that have already appeared in the main paper have not been included in the attached PDF.), it can be concluded that consistency policy in visual RL tasks faces severe policy degradation. 

In the modified revision, we expound on the correlation between the dormant ratio and the performance of the policy network in Section 3.3 as follows:

> As introduced in [3], the dormant ratio of a neural network indicates the proportion of inactive neurons and is typically used to measure the activity of the network. A higher dormant ratio implies fewer active neurons in the network, implying the network's capacity and expressiveness are damaged. In RL, the episode return is closely related to the dormant ratio of the policy network. A higher dormant ratio results in more lazy action outputs, inactive agent behavior, and lower episode returns; conversely, when policy performance is good, the policy network is usually more active, and the dormant ratio is typically lower. This phenomenon has been reported in [3-6].

Regarding the other issues raised by the reviewers, we provide detailed responses below. We strive to address all concerns raised by the reviewers. If there are any further questions, we are also very pleased to discuss them.

We have made revisions to the paper addressing certain issues. In the revised version, changes will be highlighted in blue.

[1] Boosting continuous control with consistency policy, AAMAS 2024

[2] Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning, ICLR 2024

[3] The dormant neuron phenomenon in deep reinforcement learning, ICML 2023 (Figure 9, 11, 16)

[4] DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimization, ICLR 2024(Figure 2, 11)

[5] In deep reinforcement learning, a pruned network is a good network, Arxiv 2024(Figure 11)

[6] Pretrained Visual Representations in  Reinforcement Learning, Arxiv 2024 (Figure 3, 4)

Meta-review: There is a unanimous decision for acceptance, congratulations! The sample-based entropy regularization and prioritized proximal experience regularization make sense, and the experimental comparisons are extensive and convincing on the DM and Meta-World environments.
Decision: Accept (poster)