Generalizing Consistency Policy to Visual RL with Prioritized Proximal Experience Regularization

Authors: haoran  li
zhennan  jiang
yuhui  chen
dongbin  zhao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 17:09:49

Abstract: With high-dimensional state spaces, visual reinforcement learning (RL) faces significant challenges in exploitation and exploration, resulting in low sample efficiency and training stability. As a time-efficient diffusion model, although consistency models have been validated in online state-based RL, it is still an open question whether it can be extended to visual RL. In this paper, we investigate the impact of non-stationary distribution and the actor-critic framework on consistency policy in online RL, and find that consistency policy was unstable during the training, especially in visual RL with the high-dimensional state space. To this end, we suggest sample-based entropy regularization to stabilize the policy training, and propose a consistency policy with prioritized proximal experience regularization (CP3ER) to improve sample efficiency. CP3ER achieves new state-of-the-art (SOTA) performance in 21 tasks across DeepMind control suite and Meta-world. To our knowledge, CP3ER is the first method to apply diffusion/consistency models to visual RL and demonstrates the potential of consistency models in visual RL.

Main paper: 
 introduction: 
 RL has achieved remarkable results in many fields, such as video games [1] , Go [2] , Chess [3, 4] and robotics [5] [6] [7] [8] . Since it is hard to parameterize the complex policy distribution over high-dimensional state and continuous action spaces, the performance and stability of visual RL are still unsatisfactory. As the most common policy distribution, Gaussian distribution is easy to sample, but its unimodal nature limits the expressiveness to represent complex behaviors [9] . While complex distributions have the rich, expressive power to improve the exploration ability [10] , the difficulty of sampling makes it hard to apply to online RL directly. Parameterizing the complex policy distribution to balance ease of sampling and expressiveness is a bottleneck to improving the efficiency of visual RL.As an emerging generation model, the diffusion model [11] stands out in fields such as image generation [12] [13] [14] and video generation [15, 16] with its ability to model complex distributions and ease of sampling characteristics. These properties have also been explored for learning a complex policy [17] . For example, diffusion models are used to imitat e the diverse expert policies [18, 19] or trajectories [20] [21] [22] [23] in datasets. In addition, due to their excellent expressive and data generation abilities, diffusion models are often employed to address policy constraints [24] [25] [26] and data scarcity [21, 27, 28] in offline RL. Most of these applications are limited to offline learning due to the demand for pre-collected datasets to train diffusion models.Applying diffusion models in online RL will face different problems than offline RL. Firstly, unlike pre-collected data in offline RL, the data distribution in online RL is non-stationary [29] , and it is currently unclear whether this change will impact training diffusion models. Secondly, since the optimal policy distribution is unknown, samples from this distribution are inaccessible, resulting in the ill-posed traditional score matching problem [30] . In addition, the time-inefficiency of diffusion models [31] will become more prominent with a large number of online interactions, leading to unacceptable time costs for online learning. As an efficient diffusion model, the consistency model [32] directly establishes a mapping from noise to denoised data, which is employed for online RL and achieves time efficiency and better performance [33, 34] . These methods simply replace the Gaussian model in the actor-critic framework with the consistency model and train consistency policy with the Q-loss. Although they achieve competitive performance in state-based RL tasks, this training method is incompatible with traditional score matching for diffusion models. Therefore, the question is whether this training framework is suitable for consistency model-based policy training, especially for visual RL tasks with high-dimensional state spaces.In this paper, we investigate the impact of non-stationary dataset and the actor-critic framework on consistency policy. By analyzing the dormant ratio [29] of the policy network, we find that the non-stationary of training data is not the main factor affecting the instability of consistency policy, while the Q-loss in the actor-critic framework leads to a sharp increase in the dormant ratio of the policy network, resulting in the loss of complex expression ability, which is particularly significant in visual RL tasks. To address the above issues, we suggest sample-based entropy regularization to stabilize the policy training and propose the prioritized proximal experience regularization, which uses weighted sampling to construct an appropriate proxy policy for policy regularization and achieves sample-efficiency consistency policy. Overall, our contributions are as follows:• We investigate the impact of non-stationary distribution and actor-critic framework on consistency policy in online RL, and find that the Q-loss of the actor-critic framework can impair the expressive ability of the consistency model, leading to unstable policy training. This phenomenon is particularly significant in visual RL tasks. • We suggest sample-based entropy regularization and propose a consistency policy with prioritized proximal experience regularization (CP3ER) which significantly enhances the stability of policy training with the Q-loss under the actor-critic framework. • Our proposed method performs new SOTA in 21 visual control tasks, including DeepMind control suite and Meta-world tasks. To our knowledge, our proposed CP3ER is the first method to apply diffusion/consistency models to visual RL.2 Related Work 
diffusion model in reinforcement learning: 
 Due to its high-quality sample generation ability and training stability, diffusion models [11] have been widely applied in fields such as image generation, video generation, and natural language processing and have also been promoted in RL. Since the diffusion model can represent complex distribution in datasets, it is commonly used in offline RL to model behavior policies [18, 25, 35] or expected policies [34, [36] [37] [38] to meet the requirements of diversity policy constraints and achieve a balance between constraint and exploitation. The diffusion model can also model trajectory distribution [20, 39, 40] , achieving specified trajectory generation under different guidance. In addition, diffusion models are also employed to generate data to augment limited training data [27, 28] .Although diffusion models have been widely applied in offline learning, using diffusion models in online learning remains a challenging problem. [41] proposes the concept of action gradient, which uses a value function to estimate the gradient of actions and updates the actions in the replay buffer. The diffusion model-based policy is trained based on the updated actions. [42] employs a diffusion model as the world model to generate complete rollouts at once instead of auto-regressive generation.[30] introduces the Q-score matching (QSM), which iteratively matches the parameterized score of a policy with the action gradient of its Q-function. Considering the low inference efficiency and long training time of diffusion models in RL training, [33] and [34] use consistency models instead of diffusion models and implement policy training under the actor-critic framework, achieving excellent performance in continuous control tasks.Compared to state-based RL, visual RL is faced with high-dimensional state space and continuous action space and is sensitive to training parameters and random seeds, which leads to unstable training and sample inefficiency. Image data augmentation [43] [44] [45] is a common technique to alleviate the above problems. In addition, auxilliary losses are usually combined to improve the efficiency of state representation learning from the image, such as contrastive learning loss [46] , state representation loss [47, 48] , action and state representation loss [49] , and self-supervised loss [50] . Recent works have focused on enhancing the stability of visual RL from a micro perspective of neural networks. For example, [51] proposes the visual dead trial phenomenon and introduces an adaptive regularization method for convolutional features. [52] proposes the concept of dormant neuron phenomenon to explain the behavior of the policy network during RL training. [53] controls the dormant ratio of the policy network during training so that it achieves the SOTA performance on multiple tasks. 
reinforcement learning: 
 Online RL solves sequential decision problems, typically modeled through Markov Decision Processes (MDP). MDP is represented by 6 tuples (S, A, R, T , ρ 0 , γ). Here, S is the state space, A is the action space, R and T represent the reward function and state transition function of the environment, respectively. ρ 0 is the initial distribution of the state, and γ is the discount factor. In visual RL, it is difficult for agents to directly obtain the state s t from the image o t ∈ O, where O is observation space. Therefore, an image encoder f (•) is usually required, and the state is estimated from the image through this encoder. The goal of the agent is to learn an optimal policy π * and the corresponding encoder f * to maximize the expected cumulative reward E π(f (•)) [ ∞ t=0 γ t r t ] under that policy.The consistency model [32] is a new diffusion model proposed to address the time inefficiency caused by hundreds of reverse diffusion steps in diffusion models. It replaces the iterative denoising process with learned score functions in traditional diffusion models by constructing a mapping between noise and denoised samples, and directly maps any point on the probability flow ordinary differential equation (ODE) trajectory to the original data in the reverse diffusion process. Thus, it only requires a small number of steps or even one step to achieve the generation from noise to denoised data. Consistency policy [33, 34] is a new policy representation under the actor-critic framework, which replaces traditional Gaussian models with the consistency model and updates the policy by maximizing the state-action value. Consistency policy is defined asEQUATIONwhere {τ k |k ∈ [N ]} a sub-sequence of time points on the time period [ϵ, K] with τ 1 = ϵ and τ N = K. a τ k t is the noised action and a τ k t = a t + τ k z where z ∼ N (0, I) is Gaussian noise. F θ is a trainable network that takes the state s t as a condition and outputs an action of the same dimension as the input a k t . c skip (•) and c out (•) are differentiable functions such that c skip (ϵ) = 1 and c out (ϵ) = 0 to ensure consistency policy is differentiable at τ k = ϵ. ϵ is a real number close to 0. To train this policy, [33] directly applies the above policy to the actor-critic framework and updates the policy using the following the Q-loss, which is named Consistency-AC.EQUATION)where B is the replay buffer and Q ϕ is the state-action value function. Compared to diffusion-based policies, consistency policy have significant advantages in inference speed and performance in online learning tasks [33] .The expressive ability is crucial for training the policy with RL. [29] proposes the concept of dormant ratio β r , which quantifies the expression ability of a neural network by calculating the proportion of dormant neurons in the neural networks.EQUATIONwhere N l represents the number of neurons in the l-th layer. H l τ is the number of neurons in the l-th layer whose score s l i is less than τ . The score of each neuron is calculated as follows:EQUATIONHere h l i (•) is the activation function of the i-th neuron in the l-th layer. D is the distribution of the input x. In the following sections of this paper, we use the dormant ratio to evaluate the expression ability of consistency policy during the training.As introduced in [29] , the dormant ratio of a neural network indicates the proportion of inactive neurons and is typically used to measure the activity of the network. A higher dormant ratio implies fewer active neurons in the network, implying the network's capacity and expressiveness are damaged. In RL, the episode return is closely related to the dormant ratio of the policy network. A higher dormant ratio results in more lazy action outputs, inactive agent behavior, and lower episode returns; conversely, when policy performance is good, the policy network is usually more active, and the dormant ratio is typically lower. This phenomenon has been reported in [29, [53] [54] [55] .4 Is Consistency-AC Applicable to Visual RL? Does the non-stationary distribution in online RL affect the training of consistency models? Unlike offline RL, online RL does not have pre-collected datasets. The data distribution for training the policy is constantly changing with policy improvement. So, whether this non-stationarity distribution affects the training of consistency models is a question that needs to be explored. In order to investigate the impact of non-stationarity of data for consistency model training, we record the dormant ratio of the policy network during consistency model training under two different settings: online training and offline training. We employ two tasks (MuJoCo Halfcheetah and MuJoCo Walker2d) and conduct 4 random seeds for each setting. In order to eliminate the impact of Q-loss, we follow the behavior clone setting and train the consistency model with consistency loss [32] using data from offline datasets or online replay buffers. The distribution of the data in the replay buffer varies with policy improvement. The results are shown in Figure 1 . Although there is a difference in the dormant ratios between online and offline learning settings in the Halfcheetah task, the overall trend is the same. We speculate that this difference is caused by the diversity of the samples included in the dataset. For the Walker2d task, the dormant ratios are nearly the same under two different settings. Therefore, we can infer that the non-stationary distribution of online RL does not significantly affect the consistency model training.Is the actor-critic framework suitable for training consistency policy? The actor-critic framework is a highly effective policy training framework for online RL, in which the policy network achieves policy improvement by maximizing the value function. Some works [33, 34] directly apply consistency models to this framework. Although they achieve good results in RL tasks with low dimensional state spaces, whether the actor-critic training framework is compatible with consistency model training remains a question that needs further investigation. To evaluate the impact of the actor-critic framework on the training of consistency models, we compare the dormant ratios of policy When training the policy with the consistency loss, the dormant ratios of the network show a trend of first decreasing and then increasing. This means that the policy learns the distribution from the data and then overfits the distribution. When training the policy with the Q-loss, the dormant ratio of the policy network will rapidly increase and maintain a high value, which means that the policy network will quickly fall into local optima, making the policy no longer change. In addition, we can also see that when using the Q-loss, the variance of the dormant ratios is relatively large under different random seeds. When the dormant ratio is low, the policy network can iterate properly to learn good policy. Therefore, we can determine that the Q-loss under the actor-critic framework will destabilize the consistency policy training.Will high-dimensional state space exacerbate the degradation phenomenon of consistency policy? Compared to RL with low dimensional state space, training stability in visual RL is still a challenge. In order to investigate whether the degradation phenomenon of consistency policy will become more significant under visual RL tasks, we compare the dormant ratios of the policy networks with the state as input and image as input on 2 tasks (Walker-walk and Cheetah-run in DeepMind control suite) under the setting of online learning. During the training process, only the Q-loss was used. To maintain consistency in the settings, we only count the dormant ratio of the multilayer perceptron (MLP) of the policy newtork in the image-based settings. The results are shown in (c) and (d) of Figure 2 . Similar to using the state as input, in visual RL with the image as input, most of the neurons in the MLP of the policy network go dormant. Unlike the high variance of the former, the dormant ratios of consistency policy network in visual RL maintain a low variance and a high value. This indicates that there have been almost no successful trials under different random seeds. Therefore, we can infer that visual RL will exacerbate the instability of consistency policy training caused by the Q-loss under the actor-critic framework.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? 
experimental result reproducibility: 
 Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: The paper fully discloses all necessary details for reproducing the main experimental results, including comprehensive descriptions of the algorithm in section 5 and appendix B.1, experiments setup in section 6, and parameters used in appendix B.2. 
consistency policy with prioritized proximal experience regularization: 
 Consistency Policy with Entropy Regularization. To solve the problem of consistency policy quickly falling into local optima under the influence of the Q-loss, we introduce policy regularization to stabilize policy improvement. Here, we employ entropy regularization to constrain policy behavior. The objective of RL is:EQUATIONwhere π β is the proxy distribution required for policy regularization. Entropy regularization is a commonly method for stabilizing policy training in RL. When π β is a uniform distribution, the above objective is equal to maximum entropy RL, which maximizes the entropy of the policy while optimizing the return. The prerequisite for this method is to obtain the closed form of the policy distribution to calculate its entropy. However, for diffusion models or consistency models, obtaining the closed form of the policy distribution is very difficult. Thanks to the development of generative models, we can use score matching instead of solving analytic entropy in entropy regularization RL, thus achieving sample-based policy regularization. Therefore, the training loss of consistency policy with the entropy regularization is: where L c is consistency loss defined by following:EQUATIONL c (θ) = E k∼U (1,N -1),st∼B,at∼π β ,z∼N (0,I) [λ(τ k )d(π θ (s t , a τ k+1 t , τ k+1 ), πθ(s t , a τ k t , τ k )) (7)Here λ(•) is a step-dependent weight function, d(•, •) is a distance metric. Since there is no need to obtain the closed form of the proxy distribution, only the data under that distribution needs to be obtained, making the selection of proxy distribution flexible. The remaining question is how to construct a suitable proxy distribution π β .Prioritized Proximal Experience Regularization. When the proxy distribution is uniform, this method approximates the maximum entropy consistency policy (MaxEnt CP). It should be noted that the difference between the proxy distribution and the optimal policy distribution can lead to difficulty in optimizing the above objectives. When the proxy distribution is far from the optimal policy or the proxy distribution is complex, the above optimization objectives require more samples to converge to better results. To better balance training stability and sample efficiency, we propose the prioritized proximal experience regularization (PPER). Specifically, when sampling data from the replay buffer, we design sampling weight β for each data instead of sampling the data uniformly.EQUATIONwhere α is the hyperparameter and ∆t is the interval between the sample generation step and the current step. |B| is the capacity of the replay buffer. The curves with different α are shown in Figure 3 (b). In the above settings, data closer to the current step will be sampled with a higher probability, while data farther away will have a lower probability. We refer to the above sampling method as a prioritized proximal experience (PPE).For policy evaluation, we suggest a distributional value function instead of a deterministic value function to ensure the stability and accuracy of the value estimation. Precisely, we follow [56] and use a mixture of Gaussian (MoG) to model the distribution of the state-action value. When MoG is employed for value distribution, the following loss is used to update Q-function.L q (ϕ) = -E (st,st+1)∼B,at∼π θ [ 1 M M i=1 log Z (st,at) ϕ (r t (s t , a t ) + γz ′ i )],whereEQUATION(st,at) ϕ is the estimated value distribution. According to the equation ( 9), we need to sample M target Q-values z ′ i and update the value distribution. Different from [56] , we sample only one next action a t+1 instead of multiple actions to reduce the time cost and find that this simplification can achieve good experiment results. Considering the simplicity and efficiency of DrQ-v2 [57] , our proposed consistency policy with prioritized proximal experience regularization (CP3ER) is built based on DrQ-v2. The framework is shown as Figure 3 (a) . We sample the data from the replay buffer, and augment the image with the techniques in DrQ-v2. Thanks to the natural randomness of the action from consistency policy, our method no longer requires additional exploration strategies.In addition, we only used a single Q-network to estimate the mean and variance of the mixture of Gaussian instead of double Q-network. We consider prioritized proximal experience regularization when updating consistency policy and used equation (9) when training the Q-network, which differs from DrQ-v2. The complete algorithm is included in the appendix B.1.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] We have submitted the code for training as the supplementary material, and once the paper is accepted, the code will be fully open source. Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experiments: 
 In this section, we evaluate the proposed method from the following aspects: 1) Does CP3ER have performance advantages compared to current SOTA methods? 2) Can policy regularization improve the behavior of the policy? 3) What is the impact of different modules on the performance?Environment Setup. We evaluate the methods on 21 visual control tasks from DeepMind control suite [58] and Meta-world [59] . We split these tasks into three domains, including 8 medium-level tasks in the DeepMind control suite, 7 hard-level tasks in the DeepMind control suite, and 6 tasks in the Meta-world. The details of each domain are included in the appendix C.Baselines. We compare current advanced model-free visual RL methods, including DrQ-v2 [57] , ALIX [60] and TACO [61] . The more detailed results are shown in the appendix C.Medium-level tasks in DeepMind control suite. We evaluate CP3ER on 8 medium-level tasks [57] in DeepMind control suite. The results are shown in Figure 4 . From the left part of Figure 4 , it can be seen that compared to the current SOTA methods, our proposed CP3ER has achieved better sample efficiency. It should be noted that TACO uses auxiliary losses of action and state representation during training to improve sample efficiency. Moreover, our proposed CP3ER uses no additional losses or exploration strategies. On the right part of Figure 4 , we compare the mean, Interquartile Mean (IQM), median, and optimal gap of these methods. CP3ER has significant advantages in all metrics and has more minor variance. This means that CP3ER has better training stability. Hard-level tasks in DeepMind control suite. We also evaluate CP3ER on 7 challenging tasks [53, 57] in the DeepMind control suite. It should be noted that all tasks here only train 5M frames, rather than the commonly used 30M frames in other work [53, 57] . This means it is a very hard challenge. From the results on the left part of Figure 5 , it can be seen that most methods have yet to learn effective policy within 5M frames. Our proposed CP3ER surpasses the performance of all methods without relying on any additional loss or exploration strategies. Moreover, it has significant advantages on all metrics including mean, IQM, median, and optimal gap.Meta-world. We also evaluated the methods on 6 complex tasks in the Meta-world. The results are shown in Figure 6 . We record the success rates of the tasks, and all results are based on the success rates. Compared to other methods, CP3ER can quickly learn effective manipulation policy, and the success rate can reach nearly 100% in all tasks within 2M steps. By comparing the mean, IQM, median, and optimal gap of the success rates, CP3ER has a significant performance advantage with minimal variance. Action distribution analysis with toy example. In order to further explore the impact of policy regularization on the training, we borrow the 1D continuous bandit problem [9] to analyze the policy behavior. The green curve in Figure 7 (a) shows the reward function. Within a narrow range of actions, the agent receives higher rewards, while within a broader range, the agent can only receive suboptimal rewards. Therefore, the policy needs strong exploration ability to achieve the highest return. We compare Gaussian policy with entropy regularization (MaxEnt GP), Consistency-AC [33] and consistency policy with entropy regularization (MaxEnt CP), and record the action distribution during the training. As shown in Figure 7 , Consistency-AC quickly converges to the local optimal value with the Q-loss. Policy regularization ensures the diversity of action distribution during consistency policy training, preventing the policy from falling into local optima too early. Moreover, consistency policy has robust exploration compared to the Gaussian policy and achieves higher returns. We conduct ablation studies in 2 tasks to evaluate the contribution of each module in the proposed method. In addition, to analyze the impact of proxy distribution for policy regularization on performance, we also compare several candidates, including uniform distribution or behavioral policy in the replay buffer. The results are shown in Figure 9 . It is noticeable that policy regularization is crucial for consistency policy. Without policy regularization, consistency policy (CP3ER w/o PPER) makes it difficult to learn meaningful behavior in the tasks. The proxy distribution also has an impact on the performance. Using uniform distribution to regularize policies can make the policy (CP3ER w. MaxEnt) improvement difficult, resulting in low sample efficiency. Compared to using behavior distribution in the replay buffer (CP3ER w. URB), the policy (CP3ER) obtained through prioritized proximal experience sampling has a closer distribution to the current policy, making policy optimization easier and resulting in higher sample efficiency. In addition, we find that the performance of CP3ER is significantly better than the baseline (DrQ-v2), indicating that the feasible usage of consistency policy can help solve visual RL tasks.  
conclusion: 
 In this paper, we analyze the problems faced by extending consistency policy to visual RL under the actor-critic framework and discover the phenomenon of the collapse of consistency policy during training under the actor-critic framework by analyzing the dormant ratio of the neural networks. To address this issue, we propose a consistency policy with prioritized proximal experience regularization (CP3ER) that effectively alleviates the training collapse problem of consistency policy. The method is evaluated on 21 visual control tasks and shows significantly better sample efficiency and performance than the current SOTA methods. It is worth mentioning that, to the best of our knowledge, our proposed CP3ER is the first method to apply diffusion/consistency models to visual RL tasks.Our experimental results show that the consistency policy benefits from its expressive ability and ease of sampling, effectively balancing exploration and exploitation in RL with high-dimensional state space and continuous action space. It achieves significant performance advantages without any auxiliary loss and additional exploration strategies. We believe that consistency policy will play an essential role in visual RL. There are still some issues worth exploring in future work. Firstly, auxiliary losses for representation in current visual RL have the potential to improve the performance of consistency policy. Secondly, the diversity of behavior in consistency policy is crucial for RL exploration. This paper only discusses the stability of policy training and does not analyze the diversity of behavior during training, which will help improve the exploration performance of policies.In addition, consistency policy under the on-policy framework is also a direction worth exploring. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Our research is mainly focused on visual RL, with the main potential application direction being robot skill learning. Therefore, this paper should not have the aforementioned risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have cited relevant papers and provided explanations for the referenced code in our codes. The majority of our code is licensed under the MIT license, however portions of the project are available under separate license terms: DeepMind is licensed under the Apache 2.0 license.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]Justification: We have included a README file in the provided code, which provides detailed instructions on deployment, training, and license.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
 
 Appendix: To demonstrate the phenomenon of policy degradation in Consistency-AC, we conduct experiments with different settings on more tasks and analyze the effect on the dormant ratio. For analyzing the impact of non-stationary data distribution on consistency model training in online RL, we employ 3 classic tasks from D4RL. The results are shown in Figure 10 . It can be seen that the non-stationary distribution caused by online training does not significantly affect the dormant ratio of the policy network. For analyzing the impact of the loss function and high-dimensional state input on the consistency policy, we conduct experiments on 5 tasks separately. Among them, the results of 2 tasks are presented in the main part, and the results of the remaining 3 tasks are shown here.Figure 11 shows the impact of different loss functions on the performance and the dormant ratio of consistency policy. Compared to SAC policy, consistency policy with Q-loss achieves the higher dormant ratio and the worse performance. Especially on the Finger-turn-hard task, the policy under the Consistency-AC framework hardly learn any meaningful behavior, and the dormant ratios remain at a high value. Figure 12 shows the impact of different observation inputs on Consistency-AC. Compared to state-based settings, image-based policy is more difficult to learn meaningful behavior, and its dormant ratio also maintains a higher value. In this paper, we propose Consistency Policy with Prioritized Proximal Experience Regularization (CP3ER), which is built on the basis of DrQ-v2 and its framework is shown in Figure 13 . Compared to DrQ-v2, its difference lies in the use of Prioritized Proximal Experience (PPE) when sampling data from the replay buffer, and the use of consistency policy instead of Gaussian policy in the We have demonstrated the complete procedure of CP3ER in Algorithm 1, 2 and 3.We present a summary of all the hyperparameters for CP3ER in ) h t , h t+1 ← f ξ (aug(o t )), f ξ (aug(o t+1 )) a t+1 ← π θ (o t+1 ) Compute L q (ϕ) using Equation (9) ξ ← ξ -α∇ ξ L q,ξ (ϕ) ϕ ← ϕ -α∇ ϕ L q,ξ (ϕ) φ ← (1 -τ ) φ + τ φ end procedurelearning rate and feature dimension are different, while other parameters are the same for all tasks. The parameters of our proposed method are not task-sensitive, which helps it be applied to a wider range of visual control tasks without the need for fine-tuning of parameters.In this section, we present more detailed experimental results on tasks in DeepMind control suite and Meta-world, including performance curves during the training, performance profiles, and probability of performance improvement. We compare CP3ER to 4 baselines including DrQ-v2 [57] , ALIX [60] , TACO [61] and DrM [53] . All evaluations are based on a single NVIDIA GeForce RTX 2080 Ti. For CP3ER, training a run with 2M frames on this device will take about 13 hours.• DrQ-v2: https://github.com/facebookresearch/drqv2 In this subsection, we show the detail results on the 8 medium-level tasks [57] in DeepMind control suite. These tasks include: acrobot-swingup, cheetah-run, finger-turn-hard, hopper-hop, quadrupedrun, quadruped-walk, reacher-hard and walker-walk. From the results in Figure 14 , it can be seen that our proposed CP3ER has better sample efficiency and performance on almost all tasks, even though it does not use any loss for representation learning. In addition to the performance curve during the training, we also demonstrate the performance profiles at different checkpoints and the probability of performance improvement. According to the results in Figure 15 , CP3ER is significantly superior to other baseline methods. In this subsection, we show the detail results on the 7 hard-level tasks [53, 57] in DeepMind control suite. These tasks include: dog-run, dog-stand, dog-trot, dog-walk, humanoid-run, humanoid-stand and humanoid-walk. In addition to the 3 comparison baselines considered in the previous subsection, we also consider DrM which achieves SOTA performance on hard-level tasks in DeepMind control suite by weight perturbation and exploration strategies. Figure 16 shows the results CP3ER against several baselines. It is difficult for DrQ-v2, ALIX, and TACO to learn meaningful policy within the 5M framework on these tasks. CP3ER achieves comparable performance to DrM on 4 tasks: dog-run, dog-stand, dog-trot, and dog-walk without any exploration strategy. In addition, CP3ER significantly outperforms DrM in another 3 more challenging tasks. From the results in Figure 16 , CP3ER also significantly outperforms other baselines on the tasks. Compared to the SOTA method DrM for difficult tasks, CP3ER also has a performance improvement probability of over 70%. In this subsection, we show the detail results on the 6 hard tasks [53] in Metat-world. These tasks include: assembly, disassemble, hammer, hand insert, pick place wall and stick pull. In these tasks, we compared 4 baselines, including DrM. It should be noted that in [53] , DrM achieves SOTA performance on these hard tasks in Meta-world, but we cannot reproduce the corresponding performance based on the official codes. Therefore, this result is used for reference and may cannot represent the true performance of DrM. According to the results in Figure 18 and Figure 19 , CP3ER achieves SOTA performance on all 6 hard-level tasks in Meta-world, and compared to the baselines, CP3ER has a significant performance advantage. Sample efficiency has always been a highly concerned issue in visual RL, and high expressive power of the policy can improve exploration efficiency and thus increase sample efficiency. After introducing the diffusion model into visual RL, we find a serious phenomenon of policy degradation, which results in disastrous performance. Therefore, we propose CP3ER to stabilize the training of consistent policy. In addition to visual RL tasks, CP3ER theoretically has the potential to be applied to state-based RL tasks. According to the experimental setup in [34] , we conduct a comparison to state-based RL SOTA methods, and the experimental results are shown in the Table 2 . It can be seen that compared to the current SOTA methods, CP3ER also has significant advantages in the tasks, which also proves the generalization of CP3ER on different observation tasks. In addition, we also compare the existing methods based on diffusion/consistency models following the settings in [33] , and the results are shown in Table 3 . It can be seen that compared to existing methods based on diffusion/consistency models, our method CP3ER, although aimed at visual RL problems, still has significant advantages when extended to state-based RL tasks.Halfcheetah-m 5745.9±388. Although CP3ER enhances the training stability of consistency policy in the actor-critic framework and achieves excellent performance in visual control tasks, the policy diversity of CP3ER has not been thoroughly explored. In CP3ER, this diversity only depends on the multimodal actions in the replay buffer, which gradually disappears as the training steps increases. Therefore, CP3ER may face the risk of losing diversity in consistent policy, thereby weakening its exploration ability to a certain extent. In addition, there is a lack of theoretical analysis on CP3ER. Although it is based on the actor-critic framework, its policy improvement and convergence property require more rigorous theoretical analysis.This work mainly focuses on the field of visual RL and proposes a new method that may significantly improve the efficiency of visual RL. This method may improve the efficiency of robot skill learning and have a wide impact on visual control fields such as robots, but it will not involve ethical and safety issues. Therefore, this work should not have negative social impacts.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Justification: There are no assumptions or proofs presented in this paper.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.• If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 


Decision: There is a unanimous decision for acceptance, congratulations! The sample-based entropy regularization and prioritized proximal experience regularization make sense, and the experimental comparisons are extensive and convincing on the DM and Meta-World environments.

Reviews: review: paper_summary: paper_summary: The paper analyzes the problems faced by extending consistency policy to visual RL under the actor-critic framework and discovers the phenomenon of the collapse of consistency policy during training under the actor-critic framework by analyzing the dormant rate of the neural networks. The authors propose a consistency policy with prioritized proximal experience regularization (CP3ER), which employs entropy regularization to constrain policy behavior. The experiments evaluate CP3ER with recent baselines (DrQ-v2, ALIX, TACO), on 21 visual control tasks.
strength_weakness: strength_weakness: - The method is not directly applicable to domains with discrete action spaces.
- The experiments do not compare with other consistency RL methods.
questions: questions: - According to [1], dormant rates in reinforcement learning and supervised learning represent different meanings. Can you explain in detail how dormant rates affect the expression ability of the consistency policy during training? For example, what does a high dormant rate indicate? What does an increasing dormant rate signify? And why?
    - [1] Sokar, Ghada, et al. "The dormant neuron phenomenon in deep reinforcement learning." *International Conference on Machine Learning*. PMLR, 2023.
- In Sec. 4, Fig. 3, since Q-loss under the actor-critic framework will destabilize the consistency policy training, why was only the Q-loss used during the training process?
- How do the computational cost and runtime compare with other baselines?
limitations: limitations: See the weaknesses and questions section.
paper_summary: paper_summary: The paper addresses challenges in visual RL with high-dimensional state spaces, specifically focusing on improving sample efficiency and training stability. It introduces a novel method called CP3ER, which incorporates sample-based entropy regularization and prioritized proximal experience regularization to stabilize policy training and enhance sample efficiency. The proposed CP3ER achieves state-of-the-art performance in 21 tasks across the DeepMind control suite and Meta-world, demonstrating the effectiveness of applying consistency models to visual RL. The paper identifies and addresses the instability issues caused by the Q-loss in the actor-critic framework and the non-stationary distribution of online RL data. Experimental results show that CP3ER outperforms existing methods without relying on additional exploration strategies or auxiliary losses.
strength_weakness: strength_weakness: In general, the paper is well-written and easy to follow. However, there is no illustration or theoretical support for using Eq. (8) for sampling weight. It would be better for the authors to consider more theoretical analysis on the solution design.
questions: questions: Why using eq. (8) for sampling weight? Can other functions that shares a similar look demonstrated in Fig 4(b) also work?
limitations: limitations: Limitations have been discussed in the paper.
paper_summary: paper_summary: This paper proposes a novel method that employs a consistency model for policy parameterization in online visual reinforcement learning settings. It initially identifies the issues of prior consistency actor-critic methods in high-dimensional observation online RL settings through the lens of dormant ratio. Subsequently, it proposes to incorporate entropy regularization alongside a prioritized sampling strategy to regularize the policy. Empirical experiments on 21 tasks across both the DeepMind Control Suite and MetaWorld demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: - Lack of baseline comparison in the experiments
- The scope of the paper is limited to online and visual deep reinforcement learning
- The proposed modification is straightforward compared with existing algorithms.
questions: questions: - Although CP3ER is mainly designed for online visual RL, the proposed modifications (entropy regularization, prioritized replay, Gaussian mixture distribution of value functions) do not seem to be limited to visual observation settings. It would be interesting to see how the proposed algorithm performs in state-based RL settings.
- For the 8 hard DeepMind Control Suite tasks and also Metaworld tasks, the authors compare CP3ER with TACO/ALIX/DrQ-v2 but not with DrM, which was reported in the original DrM paper to achieve good empirical performance on these tasks. This comparison here is missing.
- For the analysis (Figures 2, 3, 9), it would be beneficial to include the episodic return in the plots so that readers can understand the numerical performance of the policy in addition to the policy network’s dormant ratio, which is an intrinsic factor not necessarily correlated with the policy’s episodic return.
- The conclusion drawn from lines 202-203, “Therefore, we can infer that visual RL will exacerbate the instability of consistency policy training caused by the Q-loss under the actor-critic framework,” does not seem convincing. Even in ordinary policy
limitations: limitations: NA
paper_summary: paper_summary: The paper presents a novel method using consistency models as a policy parameterization to address online reinforcement learning. The paper first uses the dormant ratio metric to study properties of the consistency-AC method and determine that it is suitable for online RL. Then, it presents the CP3ER method, which adds an entropy regularization term to consistency AC in order to make it suitable for online RL. Finally the paper presents results showing strong performance in image-based RL benchmarks.
strength_weakness: strength_weakness: - I think the biggest issue with the paper is that the story is a bit muddled. The core contribution of the paper seems to be CP3ER and the PPER regularization method. These things should be able to work for both low-dimensional and image-based observations. However, the authors specifically state that their goal is to address image-based observations. Why? Does this algorithm no longer show any benefit in these settings? The algorithm doesn't have any contributions that specifically focus on addressing image-based observations so I don't understand why that is a central focus of the evaluation.
    - In order to properly contextualize this algorithm I think it would be necessary to compare it to low-dim baselines like SAC and PPO on a low-dim version of the benchmarks in addition to the state-based versions.
 - In the introduction the authors mention that generative models like diffusion and consistency are helpful for RL because they allow the policy to represent complex behaviors. However, none of the tasks in the training suite are 'complex' enough that they couldn't be solved by a simple MLP policy. It would be nice to see perhaps some tasks where a policy that can sample multimodal actions shines above MLP-based policies.
 - Section 4 is confusing and there are several other minor presentation issues that I explain in the 'Questions' section
questions: questions: Random Questions / Comments
 - I find the analysis related to figure 1 to be incomplete and hard to follow. 
    - First, it would be helpful to explain somewhere in your paper what we can infer about a neural network based on its dormant ratio, which would make this experiment easier to understand. Your paper seems to assume the reader is very familiar with Sokar et. al. but this need not be the case
    - How big are the datasets? Where does the online dataset come from (ie what RL algorithm, how long was it trained, how often do you update the online dataset, etc)?
    - Why is there a difference between online and offline dormant ratio for halfcheetah and not walker2d? And how can you draw conclusions from this experiment when you only run two environments that seem to get different results?
 - Figure 2 experiment is also missing a lot of details
    - What AC algorithm does 'Q-loss' refer to?
    - Where does the data for this experiment come from?
    - Once again, I'm having trouble understanding how to interpret these plots or how you come to your conclusion.
 - Figure 3 has similar issues to the last two
 - The results as they are look nice but I have a few comments
    - I think the set of baselines is incomplete. It would be nice to see a SOTA on-policy method like PPO
    - All of the results show normalized scores, but you never explain how they are normalized, which makes them impossible to compare to other papers

Nit picks:
 - It would be nice if you could add one sentence in English describing what a dormant neuron is to section 3.3 for those who are not familiar with it. It would also be nice to explain why dormant ratio is important for the expressive capabilities of a network
 - Line 134: newtork
limitations: limitations: The authors address limitations and broader impacts of their work in the appendix.

Comments: title: Most concerns addressed
comment: I appreciate the authors' clarifications and still remain in favor of acceptance.
We hope the reply can address your concerns. We sincerely appreciate your recognition of our contribution and vote to accept our work!
Thank you again for your time in helping us improve our work. We hope the reply can address your concerns. We sincerely appreciate your recognition of our contribution and vote to accept our work!
We hope the reply can address your concerns. We sincerely appreciate your recognition of our contribution and vote to accept our work!
Thank you for recognizing our work. In the introduction, we mentioned that unimodal policy such as Gaussian policy cannot model complex behavior. The complex behavior here refers to the complex exploration behavior caused by multi-modal reward landscapes. This situation exists in many tasks, which is one of the reasons why stochastic policy is needed. Previous works[1][2] have also shown that compared to unimodal policy distributions, multimodal distributions can escape from local optima easily, thereby achieving better sample efficiency and performance. Section 6.2.1 of the paper demonstrated the Gaussian distribution's limitations of exploration in this scenario with the 1D continuous bandit problem[2]. Figure 8 shows that due to the expressiveness to represent complex behavior, CP3ER achieves diverse exploration behaviors and better performance.

In addition, experimental results(Section 6.1.1, Figure 6, and Section B.2 in Appendix, Figure 14 and 15.) on hard tasks of DeepMind control suite showed that the baseline DrQ-v2 using Gaussian policy hardly learned meaningful behavior within 5M steps, while CP3ER achieved a relatively good performance, indicating that in high-dimensional continuous action spaces, CP3ER has stronger exploration ability compared to Gaussian policy.

[1] Reinforcement Learning with Deep Energy-Based Policies, ICML 2017

[2] Reparameterized Policy Learning for Multimodal Trajectory Optimization, ICML 2023
Thanks for running more experiments and addressing most of my concerns. I'll increase my score to a 5.

My one major remaining concern is the following from my original review:
> In the introduction the authors mention that generative models like diffusion and consistency are helpful for RL because they allow the policy to represent complex behaviors. However, none of the tasks in the training suite are 'complex' enough that they couldn't be solved by a simple MLP policy. It would be nice to see perhaps some tasks where a policy that can sample multimodal actions shines above MLP-based policies.

The authors pointed out that their policy is indeed still an MLP which indicates to me that I may have worded my complaint in a confusing manner. Here when I say 'MLP policy' I'm referring to a setup that directly maps an input state to a deterministic action, Gaussian distribution, or something similar, and examples of algorithms that use such policies would be vanilla SAC and DDPG. On the other hand, the authors use a consistency policy which, while still parameterized by an MLP, uses the consistency model framework in order to efficiently simulate the reverse diffusion process. The authors motivate this decision in the intro when they argue that consistency policies are more capable of addressing complex tasks that would not be solvable with a Gaussian policy. However, none of the tasks in the paper are too complex to be solved by a Gaussian, so I feel that it doesn't make sense to claim this setup is more capable of such tasks.