Title: Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churn
Venue: NeurIPS 2024
Accepted: yes
Keywords: Reinforcement Learning

Deep Learning

Regularization and Optimization
Abstract: Deep neural networks provide Reinforcement Learning (RL) powerful function approximators to address large-scale decision-making problems. However, these approximators introduce challenges due to the non-stationary nature of RL training. One source of the challenges in RL is that output predictions can churn, leading to uncontrolled changes after each batch update for states not included in the batch. Although such a churn phenomenon exists in each step of network training, it remains under-explored on how churn occurs and impacts RL. In this work, we start by characterizing churn in a view of Generalized Policy Iteration with function approximation, and we discover a chain effect of churn that leads to a cycle where the churns in value estimation and policy improvement compound and bias the learning dynamics throughout the iteration. Further, we concretize the study and focus on the learning issues caused by the chain effect in different settings, including greedy action deviation in value-based methods, trust region violation in proximal policy optimization, and dual bias of policy value in actor-critic methods. We then propose a method to reduce the chain effect across different settings, called Churn Approximated ReductIoN (CHAIN), which can be easily plugged into most existing DRL algorithms. Our experiments demonstrate the effectiveness of our method in both reducing churn and improving learning performance across online and offline, value-based and policy-based RL settings.

Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churn

Abstract

Deep neural networks provide Reinforcement Learning (RL) powerful function approximators to address large-scale decision-making problems. However, these approximators introduce challenges due to the non-stationary nature of RL training. One source of the challenges in RL is that output predictions can churn , leading to uncontrolled changes after each batch update for states not included in the batch. Although such a churn phenomenon exists in each step of network training, how churn occurs and impacts RL remains under-explored. In this work, we start by characterizing churn in a view of Generalized Policy Iteration with function approximation, and we discover a chain effect of churn that leads to a cycle where the churns in value estimation and policy improvement compound and bias the learning dynamics throughout the iteration. Further, we concretize the study and focus on the learning issues caused by the chain effect in different settings, including greedy action deviation in value-based methods, trust region violation in proximal policy optimization, and dual bias of policy value in actor-critic methods. We then propose a method to reduce the chain effect across different settings, called Churn Approximated ReductIoN ( CHAIN ), which can be easily plugged into most existing DRL algorithms. Our experiments demonstrate the effectiveness of our method in both reducing churn and improving learning performance across online and offline, value-based and policy-based RL settings, as well as a scaling setting.

1 Introduction

One fundamental recipe for the success of Deep Reinforcement Learning (DRL) is powerful approximation and generalization provided by deep neural networks, which augments the ability of RL with tabular or linear approximation to large state spaces. However, on the other side of this benefit is less control over the function dynamics. Network outputs can change indirectly to unexpected values after any random batch update for input data not included in the batch, called churn in this paper. This change is particularly problematic for an RL agent due to its non-stationary nature, which can exacerbate instability, suboptimality, and even collapse. Therefore, it is important to understand and control these undesired dynamics to address learning issues and improve performance.

Consistent efforts have been devoted by the RL community to gain a better understanding of the learning dynamics from different perspectives [Achiam et al., 2019, Kumar et al., 2022, Liu et al., 2023, Lyle et al., 2022b]. Recently, Schaul et al. [2022] studied a novel churn phenomenon in the learning process of typical value-based RL algorithms like DoubleDQN [van Hasselt et al., 2016]. The phenomenon reveals that the greedy actions of about 10% of states in the replay buffer change after a single regular batch update. Such a dramatic churn can persist throughout the learning process of DoubleDQN, causing instabilities.

In this paper, we aim to take a step further to understand how churn occurs and influences learning in different DRL settings beyond value-based RL, as well as to propose a method to control churn. We start by formally characterizing churn in view of Generalized Policy Iteration (GPI) with function approximation to best cover most DRL settings. The impact of churn is two-fold in this view: the churn in policy improvement (called the policy churn ) changes policy outputs on states that are not directly updated, while the churn in value estimation (called the value churn ) also changes the action-value landscape, thus altering greedy action and action gradient. We then discover a chain effect of churn that exhibits a cycle where the two types of churn compound and bias the learning dynamics throughout the iteration. Further, we move on from the general analysis to concrete DRL settings. We focus on the learning issues caused by the chain effect including greedy action deviation in value-based methods, trust region violation in proximal policy optimization [Schulman et al., 2017] and dual bias of policy value in actor-critic methods. The connection between the chain effect of churn and the issues necessitates an explicit control of the churn.

To this end, we propose a method called Churn Approximated ReductIoN ( CHAIN ) to reduce the chain effect of churn across different DRL settings. The main idea of CHAIN is to reduce the undesirable changes to the outputs of the policy and value networks for states (and actions) outside of the current batch of data for regular DRL training. This reduction is achieved by minimizing the change in target values for a separate batch of data when optimizing the original policy or value learning objective. CHAIN is easy to implement and plug in most existing DRL algorithms with only a few lines of code 1 . In our experiments, we evaluate the efficacy of CHAIN in a range of environments, including MinAtar [Young and Tian, 2019], OpenAI MuJoCo [Brockman et al., 2016], DeepMind Control Suite [Tassa et al., 2018] and D4RL [Fu et al., 2020]. The results show that our method can effectively reduce churn and mitigate the learning issues, thus improving sample efficiency or final performance across online and offline, value-based and policy-based DRL settings. Moreover, our results also show that our method helps to scale DRL agents up and achieves significantly better learning performance when using wider or deeper networks.

The main contributions of this work are summarized as follows: (1) We study how churn occurs and influences learning from the perspective of GPI with function approximation and present the chain effect of churn. (2) We show how churn results in three learning issues in typical DRL settings. (3) We propose a simple and general method and demonstrate its effectiveness in reducing churn and improving learning performance across various DRL settings and environments.

2 Prior Work

In the past decade, a significant effort has been made to understand the learning issues of DRL agents and propose improvements that make DRL more stable and effective. The early stage of this effort studied bias control for value approximation with deep neural networks introducing many improvements after DQN [Mnih et al., 2015] and DDPG [Lillicrap et al., 2015] to address overestimation or underestimation for value-based methods [van Hasselt et al., 2016, Bellemare et al., 2017, Hessel et al., 2018] and deep AC methods [Fujimoto et al., 2018, Haarnoja et al., 2018, Lan et al., 2020, Kuznetsov et al., 2020, Chen et al., 2021] respectively. Additional works dig deeper to diagnose the learning issues regarding instability and generalization, related to the Deadly Triad in DRL [van Hasselt et al., 2018, Achiam et al., 2019], stabilizing effect of target network [Zhang et al., 2021b, Chen et al., 2022, Piché et al., 2022], difficulty of experience replay [Schaul et al., 2016, Kumar et al., 2020, Ostrovski et al., 2021], over-generalization [Ghiassian et al., 2020, Pan et al., 2021, Yang et al., 2022], representations in DRL [Zhang et al., 2021a, Li et al., 2022, Tang et al., 2022], delusional bias [Lu et al., 2018, Su et al., 2020], off-policy correction [Nachum et al., 2019, Zhang et al., 2020, Lee et al., 2021], interference [Cobbe et al., 2021, Raileanu and Fergus, 2021, Bengio et al., 2020] and architecture choices [Ota et al., 2020].

One notable thread is to understand the learning dynamics of DRL agents with a focus on the non-stationary nature of RL. A prominent phenomenon of representation ability loss is studied in [Dabney et al., 2021, Igl et al., 2021, Kumar et al., 2021, 2022, Ma et al., 2023], which reveals how representations become less useful in later stages of learning, leading to myopic convergence. Further, empirical studies in [Nikishin et al., 2022, D'Oro et al., 2023, Sokar et al., 2023, Nauman et al., 2024] demonstrate that the loss of approximation ability becomes severe and leads to collapse when high

replay-ratios are adopted for better sample efficiency, while network resets and normalization methods can be simple and effective remedies. This is further identified as plasticity loss in DRL [Lyle et al., 2022a, Abbas et al., 2023, Dohare et al., 2023, Lyle et al., 2024, Xu et al., 2024].

Recently, it has been found that there is a dramatic change in the policy distribution where a large portion of the greedy actions change after each batch update, called policy churn Schaul et al. [2022]. Although intuitively related to generalization [Bengio et al., 2020] and interference [Liu et al., 2023], it presents a lack of understanding of churn's effect on the learning behaviors of DRL agents regarding stability, convergence, exploration, etc. Kapturowski et al. [2023] takes the inspiration and proposes a method to robustify the agent's behavior by adding an additional policy head to the value network that fits the ϵ -greedy policy via policy distillation. In this work, we further the study of churn in a more general formal framework, where churn occurs in both value and policy learning. In particular, we focus on the dynamics of churn during the learning process and how it incurs issues in different DRL algorithms and propose a practical method to reduce churn and improve learning performance.

3 Preliminaries

Reinforcement Learning (RL) is formulated within the framework of a Markov Decision Process (MDP) ⟨S , A , P , R , γ, ρ 0 , T ⟩ , defined with the state set S , the action set A , the transition function P : S ×A → P ( S ) , the reward function R : S ×A → R , the discounted factor γ ∈ [0 , 1) , the initial state distribution ρ 0 and the horizon T . The agent interacts with the MDP by performing actions from its policy a t ∼ π ( s t ) that defines the mapping from states to actions or action distributions. The objective of an RL agent is to optimize its policy to maximize the expected discounted cumulative reward J ( π ) = E π [ ∑ T t =0 γ t r t ] , where s 0 ∼ ρ 0 ( s 0 ) , s t +1 ∼ P ( s t +1 | s t , a t ) and r t = R ( s t , a t ) . The state-action value function q π defines the expected cumulative discounted reward for all s, a ∈ S × A and the policy π , i.e., q π ( s, a ) = E π [ ∑ T t =0 γ t r t | s 0 = s, a 0 = a ] .

Policy and value functions are approximated with deep neural networks to cope with large and continuous state-action space. Conventionally, q π can be approximated by Q θ with parameters θ typically through minimizing Temporal Difference (TD) loss [Sutton and Barto, 1988], i.e., L ( θ ) = E s,a ∼ D δ θ ( s, a ) 2 where D is a replay buffer and δ θ ( s, a ) is a type of TD error. A parameterized policy π ϕ with parameters ϕ can be updated by taking the gradient of the objective, i.e., ϕ ′ ← ϕ + α ∇ ϕ J ( π ϕ ) with a step size α . Value-based methods like Deep Q -Network (DQN) [Mnih et al., 2015] trains a Q -network Q θ by minimizing L ( θ ) where δ ( s, a ) = Q θ ( s, a ) -( r + γ max a ′ Q θ -( s ′ , a ′ )) and θ -denotes the target network. For policy-based methods, TD3 [Fujimoto et al., 2018] is often used to update a deterministic policy with Deterministic Policy Gradient (DPG) theorem [Silver et al., 2014]: ∇ ϕ J ( π ϕ ) = E s ∼ D [ ∇ ϕ π ϕ ( s ) ∇ a Q θ ( s, a ) | a = π ϕ ( s ) ] ; Soft Actor-Critic (SAC) [Haarnoja et al., 2018] learns a stochastic policy with the gradient: ∇ ϕ ˆ J ( π ϕ ) = E s ∼ D [ ∇ ϕ log π ϕ ( a | s )+( ∇ a log π ϕ ( a | s )) -∇ a Q θ ( s, a )) ∇ ϕ f ϕ ( ϵ ; s )) | a = f ϕ ( ϵ ; s ) ] , with noise ϵ and implicit function f ϕ for re-parameterization.

4 A Chain Effect of Value and Policy Churn

In this section, we present a formal study on value and policy churn and their impact on learning. We first introduce an intuitive overview of how churn is involved in DRL (Section 4.1). Then, we propose the definitions of the value and policy churn (Section 4.2), followed by a chain effect that reveals how the churns interplay and bias parameter update (Section 4.3).

4.1 Generalized Policy Iteration under Churn

Generalized Policy Iteration (GPI) [Sutton and Barto, 1988] is widely used to refer to the general principle of learning in an Evaluation-Improvement iteration manner, which applies to almost all RL methods. In the context of DRL, i.e., with network representation and mini-batch training, the value and policy networks' outputs can have unexpected changes, i.e., the churn , after each mini-batch training for the states not included in the batch. Such churns are neglected in most DRL methods, let alone their influence on the practical learning process. In Figure 1, we extend the classic GPI diagram by considering churn to show how it is involved in the learning process intuitively.

In the evaluation process, the parameterized Q -network Q θ approximates the value of the current policy via repeated mini-batch training. Under the impact of churn, the Q -network is not likely to

have output predictions the same as what was updated with explicit mini-batch training. For example, let's imagine we have a virtual network ¯ Q θ that only accepts the changes for the states updated by mini-batch training directly and remains unchanged for the others.

̸

Thus, the value churn , denoted by ¯ Q θ ⇝ Q θ , is an implicit process that alters the virtual network ¯ Q θ to the approximation Q θ we obtained in practice. Similarly, the policy churn ¯ π ϕ ⇝ π ϕ occurs in the improvement process. As illustrated in Figure 1, the value churn and the policy churn are interwoven in the Evaluation-Improvement process. Usually, we can assume that the churns make non-negligible changes, i.e., Q θ = ¯ Q θ and π ϕ = ¯ π ϕ . Therefore, we delve into the cause of churn, its impact on learning, and possible remedies to mitigate its negative impact in the following sections.

4.2 Definition of Value and Policy Churn

A deep neural network can have the form f θ : X → Y with parameters θ . The network is optimized for a set of input data B train = { x i } with a loss function, leading to a parameter update of θ → θ ′ . Given a reference set of input data B ref = { ¯ x i } (where B ref ∩ B train = ∅ ) and a metric d for the output space, the churn is formally defined as:

Arguably, churn is an innate property of neural networks, and it is closely related to problems like interference [Liu et al., 2020, 2023] and catastrophic forgetting [Lan et al., 2023] in different contexts.

In this paper, we focus on the churn in Q -value network Q θ and policy network π ϕ . We then obtain the definitions of the Q -value churn ( C Q , w.r.t. θ → θ ′ ) and the policy churn ( C π , w.r.t. ϕ → ϕ ′ , using a deterministic policy for demonstration) for an arbitrary state-action pair ¯ s, ¯ a ∈ B ref as follows:

Then, the definitions regarding B ref can be generalized to the batch setting by aggregating data in B ref : C Q ( θ, θ ′ , B ref ) = 1 | B ref | ∑ ¯ s, ¯ a ∈ B ref | c Q ( θ, θ ′ , ¯ s, ¯ a ) | , C π ( ϕ, ϕ ′ , B ref ) = 1 | B ref | ∑ ¯ s ∈ B ref | c π ( ϕ, ϕ ′ , ¯ s ) | . Without loss of generality, we carry out our analysis mainly regarding ¯ s, ¯ a for clarity in the following.

(U.I) How the churns C Q , C π are caused by parameter updates First, we look into the relationship between the Q -value churn C Q , the policy churn C π and the network parameter updates ∆ θ = θ ′ -θ, ∆ ϕ = ϕ ′ -ϕ . For ∆ θ , ∆ ϕ , we use typical TD learning and DPG for demonstration: ∆ θ = α | B train | ∑ s,a ∈ B train ∇ θ Q θ ( s, a ) δ θ ( s, a ) , and ∆ ϕ = α | B train | ∑ s ∈ B train ∇ ϕ π ϕ ( s ) ∇ a Q θ ( s, a ) | a = π ϕ ( s ) .

Now we characterize C Q and C π as functions of ∆ θ , ∆ ϕ with the help of Neural Tangent Kernel (NTK) [Achiam et al., 2019]. For clarity, we use B train = { s, a } and B ref = { ¯ s, ¯ a } and abbreviate B train , B ref and step size α when context is clear. Concretely,

Eq. 2 shows that the value and policy churn are mainly determined by the kernels of the Q -network k θ and the policy network k ϕ , along with the TD error and the action gradient. This indicates that churn is determined by both the network's property itself and the learning we performed with the network.

4.3 From Single-step Interplay to The Chain Effect of Churn

In addition to the first piece of understanding (U.I) that presents how parameter updates cause the churns, we discuss how the churns affect parameter updates backward with two more pieces of understanding (U.II) and (U.III) , finally shedding light on a chain effect of churn.

̸

Evaluation

Q

θ

⇝

q

π

ϕ

Improvement

π

ϕ

⇝

greedy

(

Q

θ

)

Value

Churn

Policy

Churn

π

ϕ

¯

Q

θ

Q

θ

¯

π

ϕ

(U.II) How C Q , C π deviates action gradient and policy value First, we introduce two types of deviation derived from the value and policy churn: (1) Action Gradient Deviation ( D Q ∇ a ), the change of action gradient regarding the Q -network for states and actions that are affected by the Q -value churn C Q ; (2) Policy Value Deviation ( D π Q ), the change of Q -value due to the action change for states that are affected by policy churn C π . Formally, the two types of deviation are:

One thing to note is the two types of deviation show the interplay between the value and policy churn, as the value churn derives the deviation in policy ( c Q derive - - - → d Q ∇ a ) and the policy churn derives the deviation in value ( c π derive - - - → d π Q ), as denoted by the superscripts. This interplay between the policy and value can be shown better with the expressions below (derivation details in Appendix B):

Since the action gradient and policy value play key roles in parameter updates, the deviations in them naturally incur negative impacts on learning.

The discussion thus far is within a single parameter update. Now, we discuss the implications of these single updates towards a chain of updates to shed light on the long-term effect of churn.

(U.III) How parameter updates are biased by C Q , C π and the deviations D Q ∇ a , D π Q Let us consider a segment of two consecutive updates, denoted by ( θ -, ϕ -) → ( θ, ϕ ) → ( θ ′ , ϕ ′ ) . The churns occurred during the last update ( θ -, ϕ -) → ( θ, ϕ ) participate in the current update ( θ, ϕ ) → ( θ ′ , ϕ ′ ) about to perform. Concretely, the churns affect the following aspects: (1) Q -value estimate, (2) action selection in both TD error and policy objective, and (3) the gradient of network parameters.

From these aspects, we can deduce the difference between the parameter updates under the impact of the value and policy churn (denoted by ˜ ∆ θ , ˜ ∆ ϕ ) and the conventional ones ∆ θ , ∆ ϕ . As a result, we can find that the value and policy churn, as well as the deviations derived, introduce biases in the parameter updates. We provide the complete discussion and derivation in Appendix B.2.

The analysis on the update segment ( θ -, ϕ -) → ( θ, ϕ ) → ( θ ′ , ϕ ′ ) can be forwarded, and taking the three pieces of understanding together, we arrive at the chain effect of churn.

The Chain Effect of Churn: (U.I) Parameter updates cause the value and policy churn, (U.II) which further leads to the deviations in the action gradient and policy value; (U.III) the churns and the deviations then bias following parameter updates.

As the cycle illustrated in Figure 2, the value and policy churn and the parameter update bias accumulate and can amplify each other throughout the learning process. Intuitively, the parameter update chain could derail and fluctuate under the accumulating churns and biases, thus preventing stable and effective learning. We concretize our study on the consequences in the next section.

Parameter Update

of

Q

and Policy

Q

-value and Policy Churn

Deviation in Action Gradient and Policy Value

Cause

Derive

Influence

˜ ∆

θ

t

(

B

train

)

,

˜ ∆

ϕ

t

(

B

train

)

C

Q

(

θ

t

, θ

t

+1

,

{

¯

s,

¯

a

}

)

C

π

(

ϕ

t

, ϕ

t

+1

,

{

¯

s,

¯

a

}

)

D

Q

∇

a

(

θ

t

, θ

t

+1

,

{

s, π

ϕ

t

(

s

)

}

)

D

π

Q

(

ϕ

t

, ϕ

t

+1

,

{

s

′

, π

ϕ

t

(

s

′

)

}

)

5 Reducing Value and Policy Churn in Deep RL

In this section, we show concrete learning issues caused by churn in typical DRL scenarios (Section 5.1), followed by a simple plug-in method to reduce churn and address the issues (Section 5.2).

5.1 Consequences of the Chain Effect of Churn in Different DRL Scenarios

Since churn is involved in most DRL methods, as illustrated by Figure 1, our study focuses on several typical DRL scenarios below.

Greedy action deviation in value-based methods Value-based methods like DQN train a Q -network Q θ and compute the policy by choosing the greedy action of Q θ . A consequence of computing the action greedily is that changes in the values will directly cause changes in the action distribution [Schaul et al., 2022]. Similarly to Eq. 4, this deviation can be formalized as: D Q a ∗ ( θ, θ ′ , B ref ) = 1 | B ref | ∑ ¯ s ∈ B ref I A\{ arg max a Q θ (¯ s,a ) } ( arg max ¯ a Q θ ′ (¯ s, ¯ a ) ) . We suspect that this deviation introduces instability and hinders learning, and we focus on whether reducing churn can improve the performance of value-based methods.

̸

Trust region violation in policy gradient methods Trust region plays a critical role in many policy gradient methods for reliable and efficient policy updates. Proximal Policy Optimization (PPO) [Schulman et al., 2017] uses a clipping mechanism as a simple but effective surrogate of the trust region for TRPO [Schulman et al., 2015]: Clip ( r ( ϕ old , ϕ ) , 1 -ϵ, 1 + ϵ ) and r ( ϕ old , ϕ ) = π ϕ ( a | s ) π ϕ old ( a | s ) . With respect to policy churn, even though the PPO policy conforms to the trust region for the states in the current training batch, it could silently violate the trust region for other states, including previously updated ones. Consider the policy update ϕ → ϕ ′ , it is highly likely to have r ( ϕ, ϕ ′ ) = π ϕ ′ (¯ a | ¯ s ) π ϕ (¯ a | ¯ s ) = 1 and thus r ( ϕ old , ϕ ′ ) = π ϕ ′ (¯ a | ¯ s ) π ϕ old (¯ a | ¯ s ) = r ( ϕ old , ϕ ) r ( ϕ, ϕ ′ ) = r ( ϕ old , ϕ ) . Since we have no information about r ( ϕ, ϕ ′ ) , there is no guarantee for the trust region 1 -ϵ ≤ r ( ϕ old , ϕ ′ ) ≤ 1 + ϵ to be respected after churn. Intuitively, this implicit violation is hazardous and detrimental to learning.

̸

̸

Dual bias of policy value in Actor-Critic methods Deep AC methods interleave the training between the actor-network and the critic-network. Unlike the two scenarios above, where either the value churn or the policy churn raises a learning stability issue, we present the dual bias of policy value that stems from the bilateral effect of churn. The dual bias exists in the policy value as Q θ ′ (¯ s, π ϕ ′ (¯ s )) = Q θ (¯ s, π ϕ (¯ s )) . In the context of AC methods, the policy value is used for the target computation of the critic r t + γQ θ ′ ( s t +1 , π ϕ ′ ( s t +1 )) and the optimization objective of the actor ∇ ϕ Q θ ′ ( s, π ϕ ′ ( s )) . Thus, the dual bias steers the training of the actor and the critic.

Given these negative consequences of churn, a question is raised naturally: how can we control the level of churn to mitigate the issues without introducing complex trust regions or constraints?

5.2 A Regularization Method for Churn Reduction

In this section, we propose a regularization method to reduce value and policy churn, called Churn Approximated ReductIoN ( CHAIN ). To combat the prevalence of churn's negative influence on DRL, our method should be simple to implement and easy to use with different RL methods.

Based on the definitions of the value churn ( C Q ) and the policy churn ( C π ) in Section 4.2, we propose two corresponding loss functions L QC and L PC for churn reduction. Formally, for parameterized networks Q θ t , π ϕ t at time t and a reference batch B ref sampled from replay buffer, we have:

where d π is a policy distance metric, and we use mean square error or KL divergence for deterministic or stochastic policies. Ideally, the regularization should be imposed on the post-update network parameters θ t +1 , ϕ t +1 . Since they are not available at time t , we regularize θ t , ϕ t and use θ t -1 , ϕ t -1 as the targets for a convenient and effective surrogate.

By minimizing L QC and L PC, we can reduce the value and policy churn and suppress the chain effect further. This allows us to use the churn reduction regularization terms along with standard RL objectives and arrives at DRL with CHAIN finally:

where B train , B ref are two separate batches randomly sampled from D , and λ Q , λ π are coefficients that control the degree of regularization. CHAIN serves as a plug-in component that can be implemented with only a few lines of code modification in most DRL methods. The pseudocode is omitted here, and we refer the readers to Algorithm 1 in the Appendix.

Automatic adjustment of λ Q , λ π To alleviate the difficulty of manually selecting the regularization coefficients, we add a simple but effective method to adjust λ Q , λ π adaptively during the learning process. The key principle behind this is to keep a consistent relative scale (denoted by β ) between the churn reduction regularization terms and the original DRL objectives. More precisely, by maintaining the running means of the absolute Q loss | ¯ L Q | and the VCR term | ¯ L QC | , λ Q is computed dynamically as λ Q = β | ¯ L Q | | ¯ L QC | , which is similar for λ π . This is inspired by our empirical observations and the recent study on addressing the reward scale difference across different domains [Hafner et al., 2023].

Another thing worth noting is that CHAIN helps to mitigate the loss of plasticity via churn reduction. This connection can be established by referring to the NTK expressions in Eq. 2: reducing churn encourages k θ , k ϕ to 0 and thus prevents the empirical NTK matrix from being low-rank, which is shown to be a consistent indicator of plasticity loss [Lyle et al., 2024].

6 Experiments

In the experiments, we aim to answer the following questions: (1) How large is the value and policy churn in practice, and can our method effectively reduce churn? (2) Does our method's reduction of churn address learning issues and improve performance in terms of efficiency and episode return? (3) Does CHAIN also improve the scaling abilities of deep RL?

We organize our experiments into the four subsections below that correspond to the three DRL scenarios discussed in Section 5.1 as well as a DRL scaling setting. Our experiments include 20 online RL tasks from MinAtar, MuJoCo, DMC, and 8 offline RL datasets from D4RL, as well as 6 popular algorithms, i.e., DoubleDQN, PPO, TD3, SAC, IQL, AWAC. We provide the experimental details in Appendix C and more results in Appendix D.

6.1 Results for CHAIN DoubleDQN in MinAtar

We use DoubleDQN (DDQN) [van Hasselt et al., 2016] as the value-based method and MinAtar [Young and Tian, 2019] as the experiment environments. MinAtar is an Atari-inspired testbed for convenient evaluation and reproduction. We build our DoubleDQN based on the official MinAtar code with no change to the network structure and hyperparameters. We implement CHAIN DDQN by adding a few lines of code to apply the value churn reduction regularization in the standard training of the Q -network (Eq. 7). For CHAIN DDQN, λ Q is set to 50 for Breakout and 100 for the other tasks. For CHAIN DDQN with automatic adjustment of λ Q (denoted by the suffix 'Auto' ), the target relative loss scale β is set to 0 . 05 for all the tasks.

breakout space_invaders

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Update Num

0.02

0.04

0.06

0.08

Value Change of All Actions

DDQN

DDQN w/ Churn Reduction

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Update Num

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

Value Change of All Actions

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Update Num

0.02

0.04

0.06

0.08

0.10

0.12

0.14

0.16

Percent of Greedy Action Change

DDQN

DDQN w/ Churn Reduction

Value Change of All Actions

First, to answer Question (1), we report the value churn and the greedy action deviation of DDQN in Figure 3. Each point means the average metric across randomly sampled states and the whole learning process. As expected, we can observe that the value churn accumulates as more training updates take place, leading to the growth of greedy action deviation. With CHAIN, the churn and deviation are reduced significantly. We refer the readers to Figure 9 for more statistics on the value churn.

Further, we show the learning curves of CHAIN DDQN regarding episode return in Figure 4. We can see that CHAIN consistently achieves clear improvements over DDQN in terms of both sample efficiency and final scores, especially for Asterix and Seaquest. Moreover, CHAIN (Auto) matches or surpasses the results achieved by manual coefficients, which supports Question (2) positively. In the next subsection, we evaluate how much CHAIN can improve policy gradient-based RL algorithms.

6.2 Results for CHAIN PPO in MuJoCo and DMC

Corresponding to the second DRL scenario discussed in Section 5.1, we focus on the policy churn in Proximal Policy Optimization (PPO) [Schulman et al., 2017] and try to answer the first three questions for policy gradient-based RL algorithms. We build the experiments on the public implementation of PPO from CleanRL [Huang et al., 2022] and use the continuous control tasks in MuJoCo and breakout

0.035

0.30

0.030

0.25

0.025

0.20

0.020

0.15

0.015

0.10

0.010

0.05

0.005

Percent of Greedy Action Change

0.00

0.0

0.0

2.5

2.5

5.0

5.0

asterix space_invaders

7.5

7.5

10.0

12.5

10.0

12.5

Update Num

Update Num

15.0

15.0

17.5

17.5

20.0

20.0

Value Change of All Actions

0.010

0.25

0.008

0.20

0.006

0.15

0.004

0.10

0.05

0.002

Percent of Greedy Action Change

0.00

0.000

0.0

0.0

2.

2.

0

10

20

30

40

50

60

Timestep (5

e

4)

5

10

15

20

25

30

35

breakout

DDQN

CHAIN DDQN

CHAIN DDQN (Auto)

0

10

20

30

40

50

60

Timestep (5

e

4)

10

20

30

40

50

60

70

80

90

space_invaders

0

10

20

30

40

50

60

Timestep (5

e

4)

0

5

10

15

20

25

asterix

0

10

20

30

40

50

60

Timestep (5

e

4)

10

20

30

40

50

freeway

0

20

40

60

80

100

Timestep (5

e

4)

0

10

20

30

40

seaquest

DeepMind Control (DMC) as the environments for evaluation. Following the same principle, we implement CHAIN PPO by adding the policy churn reduction regularization to the standard PPO policy training (Eq. 6), with no other modification to the public PPO implementation.

First, to understand the level of churn, we compare PPO and CHAIN PPO with different choices of λ π in terms of policy churn and episode return. In summary, we observed that PPO also exhibits clear policy churn, and CHAIN significantly reduces it throughout learning, which answers Question (1). Figure 10 shows the details for this analysis. Note that more policy churn makes it more likely to violate the trust region as C π ∝ r ( ϕ, ϕ ′ ) discussed in Section 5.1. In turn, we also observed CHAIN PPO consistently outperforms PPO in Ant-v4 and HalfCheetah-v4 across different choices of λ π .

Further, we aim to answer Question (2) and evaluate whether CHAIN can improve the learning performance of PPO in terms of episode return. For CHAIN PPO (Auto), we set the target relative loss scale β to 0.1 for MuJoCo tasks and 0.02 for DMC tasks. The results for MuJoCo and DMC tasks are reported in Figure 5. The results show that CHAIN PPO outperforms PPO in most cases with higher sample efficiency and final episode return, often significantly. We believe that our results reveal a promising direction to improve more trust-region-based and constraint-based methods in DRL by addressing the issues caused by churn.

0

10

20

30

40

50

Timestep (4

e

4)

0

500

1000

1500

2000

2500

HalfCheetah-v4

PPO

CHAIN PPO (Auto)

0

10

20

30

40

50

Timestep (4

e

4)

500

1000

1500

2000

2500

Hopper-v4

0

10

20

30

40

50

Timestep (4

e

4)

0

500

1000

1500

2000

2500

3000

3500

4000

Walker2d-v4

0

10

20

30

40

50

Timestep (1

e

5)

0

500

1000

1500

2000

2500

3000

Ant-v4

0

10

20

30

40

50

Timestep (1

e

5)

500

1000

1500

2000

2500

3000

3500

4000

Humanoid-v4

0

10

20

30

40

50

Timestep (2

e

4)

0

200

400

600

800

dm_control/ball_in_cup-catch-v0

PPO

CHAIN PPO (Auto)

0

10

20

30

40

50

Timestep (2

e

4)

0

200

400

600

800

1000

dm_control/cartpole-balance_sparse-v0

0

10

20

30

40

50

Timestep (2

e

4)

100

150

200

250

300

350

400

dm_control/quadruped-walk-v0

0

10

20

30

40

50

Timestep (2

e

4)

100

125

150

175

200

225

250

275

dm_control/quadruped-run-v0

0

10

20

30

40

50

Timestep (2

e

4)

0

100

200

300

400

500

600

700

dm_control/finger-spin-v0

0

10

20

30

40

50

Timestep (2

e

4)

50

100

150

200

250

300

350

400

dm_control/dog-stand-v0

0

10

20

30

40

50

Timestep (2

e

4)

20

40

60

80

100

120

140

dm_control/dog-walk-v0

0

10

20

30

40

50

Timestep (2

e

4)

10

20

30

40

50

60

70

80

dm_control/dog-run-v0

0

10

20

30

40

50

Timestep (2

e

4)

20

40

60

80

dm_control/dog-trot-v0

0

10

20

30

40

50

Timestep (2

e

4)

5

10

15

20

25

30

35

dm_control/dog-fetch-v0

6.3 Results for Deep Actor-Critic Methods with CHAIN in MuJoCo and D4RL

Next, we continue our empirical study and evaluate our method for deep actor-critic (AC) methods. We separate our study into online and offline settings, as presented below.

Online AC methods We use TD3 [Fujimoto et al., 2018] and SAC [Haarnoja et al., 2018] and MuJoCo environments based on the public implementation of TD3 and SAC from CleanRL . Since AC methods are bilaterally affected by churn, we consider two variants of CHAIN, either of which only applies the value churn reduction (VCR) or the policy churn reduction (PCR).

For Question (1), we again find that both TD3 and SAC exhibit value and policy churn in all environments, and CHAIN-VCR and CHAIN-PCR effectively reduce them respectively in Figure 11 and 12. For Question (2), Figure 6 shows the evaluation results regarding episode return. We can see that CHAIN-PCR often improves the learning performance, especially for Ant-v4; in contrast, CHAIN-VCR improves slightly. We hypothesize that this is because the policy interacts with the environment directly, and the target critic-network also helps to reduce the value churn due to its delayed synchronization with the online critic.

Due to the limitation of space, we refer the readers to Appendix D.3 for more results on churn reduction, the influence of different choices of λ Q , λ π , the results of combining VCR and PCR, as well as the effect of auto-adjustment of the regularization coefficient for TD3 and SAC.

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

1000

1500

2000

2500

3000

3500

Ant-v4

TD3

CHAIN TD3 (VCR)

CHAIN TD3 (PCR)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

0

500

1000

1500

2000

2500

3000

3500

Hopper-v4

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

0

1000

2000

3000

4000

Walker2d-v4

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

2000

4000

6000

8000

10000

HalfCheetah-v4

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

1000

2000

3000

4000

5000

Ant-v4

SAC

CHAIN SAC (VCR)

CHAIN SAC (PCR)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

500

1000

1500

2000

2500

3000

3500

Hopper-v4

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

0

1000

2000

3000

4000

5000

Walker2d-v4

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

2000

4000

6000

8000

10000

HalfCheetah-v4

Offline AC methods In Offline RL, a policy is trained over a fixed dataset. We investigate if reducing churn can also improve the convergence of Offline RL. We use IQL [Kostrikov et al., 2022] with D4RL Antmaze dataset [Fu et al., 2020] and AWAC [Nair et al., 2020] with Adroit for our demonstration due to their popularity and good performance in corresponding tasks. Concretely, we use the public implementation and benchmark scores for IQL and AWAC from CORL 2 . To apply CHAIN to IQL and AWAC, we implement the regularization for value churn reduction (VCR, Eq. 7) and policy churn reduction (PCR, Eq. 8) separately by adding a couple of lines of code without any other modification. We use λ π = 1 e 3 for both CHAIN IQL (PCR) and CHAIN AWAC (PCR); λ Q = 0 . 01 for CHAIN IQL (VCR) and 0 . 1 for CHAIN AWAC (VCR) across different tasks. The results are summarized in Table 1 and 2.

We observe that both CHAIN PCR and CHAIN VCR improve the scores for IQL and AWAC in most Antmaze and Adroit tasks. We hypothesize that CHAIN suppresses churn in the training of value and policy networks, thus reducing the bias caused by churn in parameter updates. One thing here that differs from TD3 and SAC considered in the online setting is that the policy network of IQL has no impact on the training of the value networks since the value networks (i.e., Q and V ) are trained purely based on in-sample data without accessing a ′ = π ϕ ( s ′ ) . Thus, although IQL does not exhibit a chain effect explicitly, the policy and value networks of IQL still have churns, which are reduced by CHAIN in this case. We provide a further empirical study in Appendix D.4.

6.4 Scaling DRL Agents with CHAIN

It is widely known that scaling DRL agents up is challenging. Naively scaling DRL agents by widening or deepening the conventional MLP networks straightforwardly often fails and could even lead to collapse. Here, we investigate the relationship between churn and scale for DRL agents, as well as the effect of CHAIN on boosting scaling performance, to answer Question (3). We take PPO and MuJoCo tasks as the exemplary setting and scale up both the policy and value networks by a scale-up ratio within { 2 , 4 , 8 , 16 } via widening or deepening . Note that the default network architecture (i.e., when the scale-up ratio equals one) for both the policy and value networks is a two-layer MLP with 256 neurons for each layer, followed by an output layer.

As expected, we observed that the performance of PPO degraded severely as the scale-up ratio increased, as shown by the solid gray lines in Figure 7. Inspired by the prior study [Obando-Ceron et al., 2024], we found using a decreased learning rate as lr / sqrt(scale-up ratio) alleviates the degradation of PPO scaling to some degree (shown by the solid red lines). We then use the learning rate setting below by default. From the perspective of churn, we also observed that scaling PPO escalates the scale of the policy churn in PPO. More results can be found in Appendix D.5. Therefore, we then evaluate the effect of CHAIN in this scaling setting. The results are shown in Figure 7 with dashed lines. By comparing the lines in the same color, we found that CHAIN improves the learning performance of PPO across almost all scale-up ratios and the two learning rate settings.

1

2

4

8

16

Scale Up Ratio

0

500

1000

1500

2000

2500

3000

Performance (score)

Ant-v4 (2M)

PPO

PPO (w/ sqrt_lr)

CHAIN PPO

CHAIN PPO (w/ sqrt_lr)

1

2

4

8

16

Scale Up Ratio

500

1000

1500

2000

2500

3000

3500

Performance (score)

Humanoid-v4 (2M)

1

2

4

8

16

Scale Up Ratio

2000

2500

3000

3500

4000

Performance (score)

Walker2d-v4 (2M)

In addition, we extend the training horizon from 2M to 10M for Ant, Humanoid, and Walker2d. The results are reported in Table 3. For both widening or deepening cases, CHAIN helps to scale up PPO and achieves clear improvement in terms of episode return. Comparatively, scaling by widening slightly outperforms deepening, which echoes the observation in [Ota et al., 2020] to some extent.

Our results indicate that uncontrolled churn could be a possible reason for the scaling issue of DRL agents, and CHAIN improves scaling by reducing churn effectively. Though appealing, CHAIN does not fully address the scaling issue per se, and achieves sub-linear scaling on DRL agents.

7 Conclusion

In this paper, we conduct a formal study of churn in a general view and present the chain effect of value and policy churn. The chain effect indicates a compounding cycle, which biases parameter updates throughout learning. We propose an easy-to-implement method for value and policy churn reduction. Our experimental results demonstrate the effectiveness of our method in reducing churn and improving learning performance over a range of DRL environments and algorithms.

References

Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos C. Machado. Loss of plasticity in continual deep reinforcement learning. arXiv preprint, arXiv:2303.07507, 2023.

Achiam, E. Knight, and P. Abbeel. Towards characterizing divergence in deep q-learning. arXiv preprint, arXiv:1903.08894, 2019.

Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Deep reinforcement learning at the edge of the statistical precipice. NeurIPS, 2021.

Marc G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In ICML, 2017.

Emmanuel Bengio, Joelle Pineau, and Doina Precup. Interference and generalization in temporal difference learning. In ICML, 2020.

Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv preprint, arXiv:1606.01540, 2016.

Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. Randomized ensembled double q-learning: Learning fast without a model. In ICLR, 2021.

Zaiwei Chen, John-Paul Clarke, and Siva Theja Maguluri. Target network and truncation overcome the deadly triad in q-learning. arXiv preprint, arXiv:2203.02628, 2022.

Karl Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient. In ICML, 2021.

Dabney, A. Barreto, M. Rowland, R. Dadashi, J. Quan, M. G. Bellemare, and D. Silver. The value-improvement path: Towards better representations for reinforcement learning. In AAAI, 2021.

Shibhansh Dohare, J. Fernando Hernandez-Garcia, Parash Rahman, Richard S. Sutton, and A. Rupam Mahmood. Maintaining plasticity in deep continual learning. arXiv preprint, arXiv:2306.13812, 2023.

Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Marc G. Bellemare, and Aaron C. Courville. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In ICLR, 2023.

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: datasets for deep data-driven reinforcement learning. arXiv preprint, arXiv:2004.07219, 2020.

Fujimoto, H. v. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In ICML, 2018.

Sina Ghiassian, Banafsheh Rafiee, Yat Long Lo, and Adam White. Improving performance in reinforcement learning by breaking generalization in neural networks. In AAMAS, 2020.

Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In ICML, 2018.

Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy P. Lillicrap. Mastering diverse domains through world models. arXiv preprint, arXiv:2301.04104, 2023.

Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Gheshlaghi Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In AAAI, 2018.

Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1-18, 2022.

Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Boehmer, and Shimon Whiteson. Transient non-stationarity and generalisation in deep reinforcement learning. In ICLR, 2021.

Steven Kapturowski, Victor Campos, Ray Jiang, Nemanja Rakicevic, Hado van Hasselt, Charles Blundell, and Adrià Puigdomènech Badia. Human-level atari 200x faster. In ICLR, 2023.

Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In ICLR, 2022.

Kumar, R. Agarwal, D. Ghosh, and S. Levine. Implicit under-parameterization inhibits dataefficient deep reinforcement learning. In ICLR, 2021.

Kumar, R. Agarwal, T. Ma, A. Courville, G. Tucker, and S. Levine. DR3: value-based deep reinforcement learning requires explicit regularization. In ICLR, 2022.

Aviral Kumar, Abhishek Gupta, and Sergey Levine. Discor: Corrective feedback in reinforcement learning via distribution correction. In NeurIPS, 2020.

Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, and Dmitry P. Vetrov. Controlling overestimation bias with truncated mixture of continuous distributional quantile critics. In ICML, 2020.

Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White. Maxmin q-learning: Controlling the estimation bias of q-learning. In ICLR, 2020.

Qingfeng Lan, Yangchen Pan, Jun Luo, and A. Rupam Mahmood. Memory-efficient reinforcement learning with value-based knowledge consolidation. Transaction on Machine Learning Research, 2023.

Jongmin Lee, Wonseok Jeon, Byung-Jun Lee, Joelle Pineau, and Kee-Eung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In ICML, 2021.

Boyan Li, Hongyao Tang, Yan Zheng, Jianye Hao, Pengyi Li, Zhen Wang, Zhaopeng Meng, and Li Wang. Hyar: Addressing discrete-continuous action reinforcement learning via hybrid action representation. In ICLR, 2022.

P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y . Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. In ICLR, 2015.

Vincent Liu, Adam White, Hengshuai Yao, and Martha White. Towards a practical measure of interference for reinforcement learning. arXiv preprint, arXiv:2007.03807, 2020.

Vincent Liu, Han Wang, Ruo Yu Tao, Khurram Javed, Adam White, and Martha White. Measuring and mitigating interference in reinforcement learning. In ICML, 2023.

Tyler Lu, Dale Schuurmans, and Craig Boutilier. Non-delusional q-learning and value-iteration. In NeurIPS, 2018.

Lyle, M. Rowland, and W. Dabney. Understanding and preventing capacity loss in reinforcement learning. In ICLR, 2022a.

Clare Lyle, Mark Rowland, Will Dabney, Marta Kwiatkowska, and Yarin Gal. Learning dynamics and generalization in deep reinforcement learning. In ICML, 2022b.

Clare Lyle, Zeyu Zheng, Khimya Khetarpal, Hado van Hasselt, Razvan Pascanu, James Martens, and Will Dabney. Disentangling the causes of plasticity loss in neural networks. arXiv preprint, arXiv:2402.18762, 2024.

Ma, H. Tang, D. Li, and Z. Meng. Reining generalization in offline reinforcement learning via representation distinction. In NeurIPS, 2023.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.

Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint, arXiv:1912.02074, 2019.

Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint, arXiv:2006.09359, 2020.

Michal Nauman, Michal Bortkiewicz, Mateusz Ostaszewski, Piotr Milos, Tomasz Trzcinski, and Marek Cygan. Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning. arXiv preprint, arXiv:2403.00514, 2024.

Nikishin, M. Schwarzer, P. D'Oro, P. Bacon, and A. C. Courville. The primacy bias in deep reinforcement learning. In ICML, Proceedings of Machine Learning Research, 2022.

Johan S. Obando-Ceron, Aaron C. Courville, and Pablo Samuel Castro. In deep reinforcement learning, a pruned network is a good network. arXiv preprint, arXiv:2402.12479, 2024.

Georg Ostrovski, Pablo Samuel Castro, and Will Dabney. The difficulty of passive learning in deep reinforcement learning. In NeurIPS, 2021.

Kei Ota, Tomoaki Oiki, Devesh K. Jha, Toshisada Mariyama, and Daniel Nikovski. Can increasing input dimensionality improve deep reinforcement learning? In ICML, 2020.

Yangchen Pan, Kirby Banman, and Martha White. Fuzzy tiling activations: A simple approach to learning sparse representations online. In ICLR, 2021.

Alexandre Piché, Valentin Thomas, Joseph Marino, Rafael Pardinas, Gian Maria Marconi, Christopher Pal, and Mohammad Emtiyaz Khan. Bridging the gap between target networks and functional regularization. Transactions on Machine Learning Research, 2022.

Roberta Raileanu and Rob Fergus. Decoupling value and policy for generalization in reinforcement learning. In ICML, 2021.

Schaul, A. Barreto, J. Quan, and G. Ostrovski. The phenomenon of policy churn. arXiv preprint, arXiv:2206.00730, 2022.

Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In ICLR, 2016.

John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In ICML, 2015.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint, arXiv:1707.06347, 2017.

Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. A. Riedmiller. Deterministic policy gradient algorithms. In ICML, 2014.

Sokar, R. Agarwal, P. S. Castro, and U. Evci. The dormant neuron phenomenon in deep reinforcement learning. arXiv preprint, arXiv:2302.12902, 2023.

Dijia Su, Jayden Ooi, Tyler Lu, Dale Schuurmans, and Craig Boutilier. Conqur: Mitigating delusional bias in deep q-learning. In ICML, 2020.

S. Sutton and A. G. Barto. Reinforcement learning: An introduction. IEEE Transactions on Neural Networks, 16:285-286, 1988.

Tang, Z. Meng, J. Hao, C. Chen, D. Graves, D. Li, C. Yu, H. Mao, W. Liu, Y. Yang, W. Tao, and L. Wang. What about inputting policy in value function: Policy representation and policy-extended value function approximator. In AAAI, 2022.

Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. de Las Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, T. P. Lillicrap, and M. A. Riedmiller. Deepmind control suite. arXiv preprint, arXiv:1801.00690, 2018.

Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double qlearning. In AAAI, 2016.

Hado van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad. arXiv preprint, arXiv:1812.02648, 2018.

Guowei Xu, Ruijie Zheng, Yongyuan Liang, Xiyao Wang, Zhecheng Yuan, Tianying Ji, Yu Luo, Xiaoyu Liu, Jiaxin Yuan, Pu Hua, Shuzhen Li, Yanjie Ze, Hal Daumé III, Furong Huang, and Huazhe Xu. Drm: Mastering visual reinforcement learning through dormant ratio minimization. In ICLR, 2024.

Ge Yang, Anurag Ajay, and Pulkit Agrawal. Overcoming the spectral bias of neural value approximation. In ICLR, 2022.

Kenny Young and Tian Tian. Minatar: An atari-inspired testbed for more efficient reinforcement learning experiments. arXiv preprint, arXiv:1903.03176, 2019.

Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. In ICLR, 2021a.

Ruiyi Zhang, Bo Dai, Lihong Li, and Dale Schuurmans. Gendice: Generalized offline estimation of stationary values. In ICML, 2020.

Shangtong Zhang, Hengshuai Yao, and Shimon Whiteson. Breaking the deadly triad with a target network. In ICML, 2021b.

A Limitations

Our work is limited in several directions below and we expect further studies on these points in the future.

First, the theoretical analysis of the chain effect under concrete assumptions remains to be explored. Other perspectives that may influence churn, such as network structure, representation learning, and experience replay are not considered in this work, which are worthwhile to study in the future.

Moreover, although we provided a simple method to adjust the regularization coefficients dynamically throughout learning by keeping a consistent relative loss scale, it is not sufficient for us to use the same hyperparameter for different domains. We believe that using normalization techniques to unify the scales in different domains is necessary to address this point, similar to the work in [Hafner et al., 2023].

Besides, for algorithms that involve the learning of both policy and value (e.g., deep AC methods), the implementation of CHAIN in different problems faces the question of choosing the best option among PCR, VCR and DCR. For this, we expect to develop a better method to integrate the effect of PCR and VCR in the future.

Another remaining problem is the lack of an in-depth understanding of the positive and negative effects of churn on the generalization of DRL agents, which could drive new methods that better leverage the potential of churn.

B Additional Formal Analysis

B.1 The NTK Expressions of Two Types of Deviation

The NTK expression for the action gradient deviation D Q ∇ a is straightforward to obtain by plugging in the NTK expression for the Q -value churn:

And for the policy value deviation D π Q , the NTK expression is obtained by performing Taylor expansion of π ϕ (¯ s ) and plugging in the NTK expression for the policy churn:

The NTK expression of the two types of deviation above indicates that action gradient deviation and policy value deviation are mainly influenced by the second-order partial derivatives of Q θ and the deterministic policy gradient (DPG) of π ϕ at the state ¯ s considered, in addition to the parameter update. An interesting observation here is, policy value deviation of policy churn has an implicit optimization effect (while altered by ∆ ϕ ) for the reference states (i.e., ¯ s ) although the parameter update ϕ → ϕ ′ is performed for the training states (i.e., s ). We observe that D π Q is positive in overall in our empirical investigation later (Appendix D). However, careful considerations are needed in the future because the implicit optimization could be delusional as churn also occurs in Q θ through the learning process.

∼

𝑄

!

!

𝜋

"

!

𝑄

!

!"#

𝜋

"

!"#

𝒟

!

"

(𝜙

#$%

, 𝜙

#

)

𝒟

∇

!

!

(𝜃

#$%

, 𝜃

#'%

)

Δ

(

"#$

…

Δ

)

"#$

𝒞

!

(𝜃

#$%

, 𝜃

#

)

𝒞

"

(𝜙

#$%

, 𝜙

#

)

𝑄

!

!$#

Δ

(

"

𝜋

"

!$#

Δ

)

"

…

…

…

∼

…

…

∼

∼

B.2 Complete Discussions and Derivations for Section 4.3

Recall conventional gradients ∆ θ , ∆ ϕ of value network and policy network:

where δ θ ( s, a ) = r + γQ θ ( s ′ , π ϕ ( s ′ )) .

Now let us re-consider the gradients by taking into consideration the value and policy churn, denoted by ˜ ∆ θ , ˜ ∆ ϕ . Our purpose is to formulate the difference between ˜ ∆ θ , ˜ ∆ ϕ and ∆ θ , ∆ ϕ as functions of the policy and value churns, as well as their derivatives.

More derivatives of C Q , C π Before looking into the gradients ˜ ∆ θ , ˜ ∆ ϕ , we need three more definitions for network parameter gradient deviation caused by value and policy churn, and action gradient deviation caused by policy churn, during parameter update θ → θ ′ , ϕ → ϕ ′ .

Q -network Gradient Deviation of Value Churn: D Q ∇ θ ( θ, θ ′ , { ¯ s, ¯ a } ) = ∇ θ ′ Q θ ′ (¯ s, ¯ a ) -∇ θ Q θ (¯ s, ¯ a ) .

Policy Network Gradient Deviation of Policy Churn: D π ∇ ϕ ( ϕ, ϕ ′ , { ¯ s } ) = ∇ ϕ ′ π ϕ ′ (¯ s ) -∇ ϕ π ϕ (¯ s ) .

Action Gradient Deviation of Policy Churn: D π ∇ a ( ϕ, ϕ ′ , { ¯ s } ) = ∇ ¯ a ′ Q (¯ s, ¯ a ′ ) | ¯ a ′ = π ϕ ′ (¯ s ) -∇ ¯ a Q (¯ s, ¯ a ) ¯ a = π ϕ (¯ s )

Note D π ∇ a ( ϕ, ϕ ′ , { s } ) denotes the action gradient caused by the policy churn for the Q function affected by the churns Q θ ✿✿ rather than Q θ . D π ∇ a is a further consequence of policy value deviation caused by the policy churn D π Q ( ϕ -, ϕ, { s } ) .

To shed light on the long-term effect of churn, we depict a typical iterative update scenario in Figure 8, where the Q -network and policy network update with corresponding gradients in a chain. Different from conventional analysis, we explicitly consider the value and policy churn and study how they affect the chain of parameter updates.

As in Section 4.3, we focus on the segment ( θ -, ϕ -) → ( θ, ϕ ) → ( θ ′ , ϕ ′ ) on the chain of update. The churns occurred during the past update ( θ -, ϕ -) → ( θ, ϕ ) , which should further affect the update about to perform at present ( θ, ϕ ) → ( θ ′ , ϕ ′ ) . Concretely, the churns and the deviations affect the following aspects: (1) Q -value estimate and (2) action selection in both TD error and policy objective, and (3) the gradient of network parameter. Now we are ready to deduce the parameter update under the effect of the value and policy churn below. Note that we use ✿ under the terms that are affected by

value and policy churn:

As a result, we can find that the value and policy churn, as well as the deviations derived, introduce biases in the parameter updates.

Recall that the parameter updates cause the churns constantly, the analysis on the update segment ( θ -, ϕ -) → ( θ, ϕ ) → ( θ ′ , ϕ ′ ) can be forwarded and leads to the cycle illustrated in Figure 2: (1) parameter update causes the value and policy churn C Q , C π , which (2) further deviates the action gradient and policy value D π Q , D Q ∇ a (and the other deviations); (3) the churns and the deviations then bias following parameter updates with ˜ ∆ θ , ˜ ∆ ϕ . Consequently, the value and policy churn and the parameter update bias accumulate and can amplify each other throughout the learning process.

Apparently, the long-term chain effect is intricate as the training process is stochastic and the churns are influenced by various factors, e.g., network structure, learning objective, and data distribution. We leave theoretical studies under concrete assumptions in the future.

C Experimental Details

C.1 Compute Resources and Time Cost

We use Nvidia V100 GPU for our experiments. The additional computation introduced by CHAIN lies in the sampling and training with a second batch of data for the churn reduction regularization. In essence, it just increases the batch size by 2 to use CHAIN, resulting in a constant change in complexity with respect to the default DRL algorithm. This is a small price to pay to reduce the churn and increase policy performance. In practice, we also observe similar wall-clock time cost for standard DRL methods and their CHAIN versions.

More importantly, since CHAIN often brings higher sample efficiency, i.e., achieving the same level of score with fewer interaction steps, this implies that CHAIN accelerates learning and achieves good performance earlier. Concrete examples can be found in our experimental results.

C.2 Empirical Metrics for the Investigation of the Chain Effect

To investigate the extent of the value churn and the policy churn, we compare the output changes between current networks ( θ t , ϕ t ) and past network versions ( θ -, ϕ -) ∈ { ( θ t -i , ϕ t -i ) } N i =1 .

For policy-based methods, we study TD3 and SAC for MuJoCo. We compute the value churn ( ˆ C Q sa , ˆ C | Q sa | ), the policy churn ( ˆ C π ) and the value deviation of policy churn ( ˆ D π Q ) throughout learning.

For value-based methods, we compute the percentage of the greedy action deviation ( ˆ D Q a ⋆ ), the value churn of greedy action ( ˆ C Q a ⋆ ) and the value churn of all actions ( ˆ C Q s , ˆ C | Q s | ) These metrics are summarized in Table 4.

We use N = 50 and N = 20 for MuJoCo and MinAtar environments and compute the metrics at an interval of 1k parameter updates. For each type of metric in Table 4 and update number i ∈ { 1 , . . . , N } , we compute the mean of the quantities throughout learning. We refer the readers to Figure 9, 11 and 12 for the results in Appendix D.

C.3 Code Implementation

We use the public implementations of PPO, TD3 and SAC in CleanRL 4 as our codebase. The actor and critic networks are two-layer MLPs with 256 units for each layer. For DoubleDQN, we modified the DQN implementation provided in the official code of MinAtar paper 5 with no change to the network structure, recommended hyperparameter choices, etc. Hyperparameters are listed in Table 5, Table 6 and Table 7. For the experiments in the offline RL setting, we use the public implementation and benchmark scores for IQL and AWAC from CORL 6 .

One thing to note is that the data in the training batch and the regularization batch should be nonoverlapping in principle, but we found simply sampling two random batches from the replay buffer independently works well. From the probabilistic perspective, the overlap could happen at a low probability, which is determined by the batch size and the size of the replay buffer.

We make no change to the state/observation and reward of MuJoCo and MinAtar environments. No additional tricks like state normalization, reward normalization are used in our experiments.

C.4 Other Discussions

Discussion on the Data Batches Used in Training and Anlysis We use separate sets of data for churn reduction regularization (i.e., the regularization set) and churn investigation/evaluation (i.e., the actual reference set), and they are randomly sampled at each network update. In other words, if count in the regular batch for training, we have three separate batches in total: (1) a regular training

batch for standard RL loss computation, (2) a regularization batch for churn reduction loss, and (3) a reference batch for churn evaluation (optional) throughout the learning process. Note that the reference batch is only used for churn evaluation and does not influence learning.

Algorithm 1 Deep RL with Churn Approximated ReductIoN ( CHAIN ).

D Complete Results

D.1 More Empirical Analysis on the Value Churn in DoubleDQN

In Figure 9 shows the statistics of the three metrics defined in Table 4. Horizontal axes show the number of parameter updates after which the statistics are computed. Each point is obtained by averaging all the quantities throughout learning. Curves and shades denote means and standard errors across six random seeds.

We can observe that the amount of value churn ( ˆ C | Q s | ) accumulates as the update number increases. Although there does not exist an explicit policy network, the percentage of greedy action deviation ( ˆ C Q a ⋆ ) goes up to over 20% . However, different from the case for TD3 and SAC, the value of greedy action ( ˆ D Q a ⋆ ) decreases and the value churn ( ˆ C Q s ) exhibits a similar trend.

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Update Num

0.02

0.04

0.06

0.08

0.10

0.12

0.14

0.16

Percent of Greedy Action Change

breakout

DDQN

DDQN w/ Churn Reduction

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Update Num

0.00

0.05

0.10

0.15

0.20

0.25

0.30

Percent of Greedy Action Change

space_invaders

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Update Num

0.00

0.05

0.10

0.15

0.20

0.25

Percent of Greedy Action Change

asterix

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Update Num

0.000

0.025

0.050

0.075

0.100

0.125

0.150

0.175

0.200

Percent of Greedy Action Change

freeway

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Update Num

0.00

0.05

0.10

0.15

0.20

0.25

Percent of Greedy Action Change

seaquest

(a) Percentage of Greedy Action Change

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Update Num

-0.15

-0.10

-0.05

0.00

0.05

Value Change of Greedy Action (1e-2)

breakout

DDQN

DDQN w/ Churn Reduction

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Update Num

-0.4

-0.3

-0.2

-0.1

0.0

Value Change of Greedy Action (1e-2)

space_invaders

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Update Num

-0.12

-0.10

-0.08

-0.06

-0.04

-0.02

0.00

Value Change of Greedy Action (1e-2)

asterix

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Update Num

-0.04

-0.03

-0.02

-0.01

0.00

0.01

Value Change of Greedy Action (1e-2)

freeway

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Update Num

-0.02

0.00

0.02

0.04

0.06

Value Change of Greedy Action (1e-2)

seaquest

(b) Value Change of Greedy Action

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Update Num

0.02

0.04

0.06

0.08

Value Change of All Actions

breakout

DDQN

DDQN w/ Churn Reduction

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Update Num

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

Value Change of All Actions

space_invaders

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Update Num

0.005

0.010

0.015

0.020

0.025

0.030

0.035

Value Change of All Actions

asterix

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Update Num

0.000

0.002

0.004

0.006

0.008

0.010

Value Change of All Actions

freeway

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Update Num

0.005

0.010

0.015

0.020

0.025

0.030

0.035

Value Change of All Actions

seaquest

(c) Value Change of All Actions

D.2 More Results of CHAIN PPO

Figure 10 provides the learning performance of CHAIN PPO on four MuJoCo tasks. Moreover, we also report the conventional PPO policy loss and the regularization loss L PC ( ϕ ) during the learning process. Figure 10 also shows the results for different choices of the hyperparameter λ π .

From the results, we can observe: (1) PPO exhibits clear policy churn (the second column). One thing to note is that the fading of the policy churn is due to the decay of the learning rate in CleanRL 's PPO implementation. CHAIN (2) reduces the policy churn of PPO effectively, and (3) shows performance improvement in Ant and HalfCheetah and comparable performance in Hopper and Walker2d. For the hyperparameter choice, in practice, we found the churn reduction coefficient between 1e3 and 1e4 works well for Ant, while 50 works best in HalfCheetah.

We hypothesize that the choice of λ π is likely to be related to the scale difference between policy loss -J ( ϕ ) (the third column) and regularization term L PC ( ϕ ) (the fourth column). This motivates the proposal of our method for automatic adjustment of the regularization coefficient presented in Section 5.2.

0

5

10

15

20

25

30

35

40

Timestep (5

e

4)

0

500

1000

1500

2000

2500

3000

3500

PPO

CHAIN PPO (

λπ

=2000)

CHAIN PPO (

λπ

=5000)

CHAIN PPO (

λπ

=10000)

0

5

10

15

20

25

30

35

40

Timestep (5

e

4)

0.000

0.002

0.004

0.006

0.008

0.010

0.012

0

5

10

15

20

25

30

35

40

Timestep (5

e

4)

-0.09

-0.08

-0.07

-0.06

-0.05

-0.04

-0.03

-0.02

Policy Loss

0

5

10

15

20

25

30

35

40

Timestep (5

e

4)

0.000

0.001

0.002

0.003

0.004

0.005

0.006

0.007

Regularization Loss (1e-4)

(a) Ant

0

5

10

15

20

25

30

35

40

Timestep (5

e

4)

0

1000

2000

3000

4000

PPO

CHAIN PPO (

λπ

=5)

CHAIN PPO (

λπ

=50)

CHAIN PPO (

λπ

=2000)

0

5

10

15

20

25

30

35

40

Timestep (5

e

4)

0.000

0.005

0.010

0.015

0.020

0

5

10

15

20

25

30

35

40

Timestep (5

e

4)

-0.05

-0.04

-0.03

-0.02

-0.01

Policy Loss

0

5

10

15

20

25

30

35

40

Timestep (5

e

4)

0.00

0.01

0.02

0.03

0.04

0.05

0.06

Regularization Loss (1e-4)

(b) HalfCheetah

0

5

10

15

20

25

30

35

40

Timestep (5

e

4)

0

500

1000

1500

2000

2500

3000

3500

4000

PPO

CHAIN PPO (

λπ

=5)

CHAIN PPO (

λπ

=50)

CHAIN PPO (

λπ

=2000)

0

5

10

15

20

25

30

35

40

Timestep (5

e

4)

0.000

0.005

0.010

0.015

0.020

0

5

10

15

20

25

30

35

40

Timestep (5

e

4)

-0.05

-0.04

-0.03

-0.02

-0.01

Policy Loss

0

5

10

15

20

25

30

35

40

Timestep (5

e

4)

0.00

0.01

0.02

0.03

0.04

0.05

0.06

Regularization Loss (1e-4)

(c) Walker2d

0

5

10

15

20

25

30

35

40

Timestep (5

e

4)

0

500

1000

1500

2000

2500

PPO

CHAIN PPO (

λπ

=5)

CHAIN PPO (

λπ

=50)

CHAIN PPO (

λπ

=2000)

0

5

10

15

20

25

30

35

40

Timestep (5

e

4)

0.000

0.005

0.010

0.015

0.020

0.025

0.030

0

5

10

15

20

25

30

35

40

Timestep (5

e

4)

-0.030

-0.025

-0.020

-0.015

-0.010

-0.005

0.000

Policy Loss

0

5

10

15

20

25

30

35

40

Timestep (5

e

4)

0.00

0.02

0.04

0.06

0.08

0.10

0.12

0.14

Regularization Loss (1e-4)

(d) Hopper

D.3 More Results of CHAIN for Deep AC Methods

We refer the readers to the following figures and tables for concrete additional results:

Figure 11 and Figure 12 show how the value churn reduction (VCR) and policy churn reduction (PCR) take effect during the learning process of TD3 and SAC. The four metrics defined in Table 4 are used.

Figure 13 and Figure 14 show the results of different choices of λ Q and λ π when VCR, PCR, or VCR+PCR (i.e., denoted by DCR) in CHAIN TD3 and CHAIN SAC.

Figure 15 shows an overall comparison among VCR, PCR, DCR when using either of them in CHAIN TD3 and CHAIN SAC. We also provide the comparison with Reliable Metrics 7 [Agarwal et al., 2021] in Figure 17 and Figure 18.

Figure 16 shows the evaluation of the automatic adjustment method of λ π for CHAIN TD3 in four MuJoCo tasks.

The Effect of CHAIN-VCR and CHAIN-PCR in Reducing Churn For TD3 and SAC, the amount of value churn ( ˆ C | Q sa | ) and policy churn ( ˆ C π ) increases as the update number i increases. This is expected as the chain effect indicates the accumulation of churn and deviation. Besides, the churn saturates after a sufficient number of updates on the networks. We hypothesize the target network helps cap the amount of churn.

Moreover, we observe a positive policy value deviation of action churn ( ˆ D π Q ), which matches our discussion on implicit optimization effect of the policy churn in Appendix B; in contrast, the value churn ( ˆ C Q sa ) fluctuates above and below 0.

According to the relative scale, TD3 and SAC show less value and policy churn than DoubleDQN. This is due to the structure independence of actor and critic networks. Another possible reason is that MuJoCo locomotion may have smoother underlying problem dynamics than the MinAtar.

̂

̂

̂

̂

0

5

10

15

20

Timestep (5

e

4)

2000

4000

6000

8000

10000

TD3

TD3 w/ VCR (

λQ

=0.5)

TD3 w/ VCR (

λQ

=1.0)

TD3 w/ PCR (

λπ

=0.1)

TD3 w/ PCR (

λπ

=1.0)

0

10

20

30

40

50

Update Num

40

60

80

100

120

Abs. Value Churn



|

Qsa

| (1e-2)

0

10

20

30

40

50

Update Num

0

1

2

3

4

Value Churn



Qsa

(1e-2)

0

10

20

30

40

50

Update Num

0.01

0.02

0.03

0.04

0.05

Policy Churn



π

0

10

20

30

40

50

Update Num

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Policy Value Deviation



π

Q

(1e-2)

(a) HalfCheetah-v4

0

5

10

15

20

Timestep (5

e

4)

0

500

1000

1500

2000

2500

3000

3500

TD3

TD3 w/ VCR (

λQ

=0.1)

TD3 w/ VCR (

λQ

=1.0)

TD3 w/ PCR (

λπ

=0.1)

TD3 w/ PCR (

λπ

=1.0)

0

10

20

30

40

50

Update Num

20

30

40

50

60

70

80

90

Abs. Value Churn



|

Qsa

| (1e-2)

0

10

20

30

40

50

Update Num

-1

0

1

2

Value Churn



Qsa

(1e-2)

0

10

20

30

40

50

Update Num

0.02

0.04

0.06

0.08

0.10

Policy Churn



π

0

10

20

30

40

50

Update Num

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

Policy Value Deviation



π

Q

(1e-2)

(b) Hopper-v4

0

5

10

15

20

Timestep (5

e

4)

0

1000

2000

3000

4000

TD3

TD3 w/ VCR (

λQ

=0.1)

TD3 w/ VCR (

λQ

=0.5)

TD3 w/ PCR (

λπ

=0.5)

TD3 w/ PCR (

λπ

=1.0)

0

10

20

30

40

50

Update Num

40

60

80

100

120

Abs. Value Churn



|

Qsa

| (1e-2)

0

10

20

30

40

50

Update Num

-1

0

1

2

3

Value Churn



Qsa

(1e-2)

0

10

20

30

40

50

Update Num

0.01

0.02

0.03

0.04

0.05

0.06

0.07

Policy Churn



π

0

10

20

30

40

50

Update Num

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Policy Value Deviation



π

Q

(1e-2)

(c) Walker2d-v4

0

5

10

15

20

Timestep (5

e

4)

1000

1500

2000

2500

3000

3500

TD3

TD3 w/ VCR (

λQ

=0.1)

TD3 w/ VCR (

λQ

=0.5)

TD3 w/ PCR (

λπ

=10)

TD3 w/ PCR (

λπ

=20)

0

10

20

30

40

50

Update Num

10

15

20

25

30

35

Abs. Value Churn



|

Qsa

| (1e-2)

0

10

20

30

40

50

Update Num

-0.5

0.0

0.5

1.0

Value Churn



Qsa

(1e-2)

0

10

20

30

40

50

Update Num

0.000

0.005

0.010

0.015

0.020

0.025

0.030

0.035

0.040

Policy Churn



π

0

10

20

30

40

50

Update Num

0.0

0.5

1.0

1.5

2.0

2.5

Policy Value Deviation



π

Q

(1e-2)

̂

̂

̂

̂

Discussion on the Effect of CHAIN-VCR, -PCR, -DCR in Improving Episode Return In addition to the two variants of CHAIN introduced in the main body of this paper, we introduce the third one, i.e., Double Churn Reduction (DCR), which corresponds to applying VCR and PCR at the same time. The related results are reported in Figure 13, 14, 15.

For CHAIN-VCR and CHAIN-PCR, we found that CHAIN-PCR often improves the learning performance, especially for Ant-v4. In contrast, CHAIN-VCR improves slightly. We hypothesize that this is because policy interacts with the environment directly and the target critic-network also helps to cap the value churn. Between TD3 and SAC, CHAIN-PCR works better for TD3 rather than SAC. We hypothesize that the variation (regarding the scale and range) is higher in optimizing the Maximum-Entropy objective and KL-based PCR term together for SAC than in optimizing the Q objective and L2-based PCR term for TD3. Another hypothesis is that the Maximum-Entropy nature of SAC prefers the encouragement of more stochasticity in policy.

For CHAIN-DCR, we found that it is not easy to gain an immediate additive improvement regarding episode return when using the same hyperparameter choices λ Q , λ π from both sides. We suggest that this reflects the intricate nature of the chain effect. As mentioned in Section 7, this points out the limitation of this work in aspects like the in-depth theoretical analysis of the long-term chain effect, the lack of automatic coefficient adjustment, and the finer-grained understanding of the positive and negative effects of churn. We leave these directions for future work.

̂

̂

̂

̂

̂

̂

̂

̂

̂

̂

̂

̂

̂

̂

̂

̂

0

5

10

15

20

Timestep (5

e

4)

2000

4000

6000

8000

10000

SAC

SAC w/ VCR (

λQ

=0.1)

SAC w/ VCR (

λQ

=1.0)

SAC w/ PCR (

λπ

=5

e

-4)

0

10

20

30

40

50

Update Num

40

60

80

100

120

Abs. Value Churn



|

Qsa

| (1e-2)

0

10

20

30

40

50

Update Num

-1

0

1

2

3

4

Value Churn



Qsa

(1e-2)

0

10

20

30

40

50

Update Num

0.0

0.2

0.4

0.6

0.8

1.0

Policy Churn (KL)



π

0

10

20

30

40

50

Update Num

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Policy Value Deviation



π

Q

(1e-2)

(a) HalfCheetah-v4

0

5

10

15

20

Timestep (5

e

4)

500

1000

1500

2000

2500

3000

3500

SAC

SAC w/ VCR (

λQ

=0.5)

SAC w/ VCR (

λQ

=1.0)

SAC w/ PCR (

λπ

=1

e

-4)

SAC w/ PCR (

λπ

=5

e

-4)

0

10

20

30

40

50

Update Num

20

30

40

50

60

70

80

Abs. Value Churn



|

Qsa

| (1e-2)

0

10

20

30

40

50

Update Num

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

Value Churn



Qsa

(1e-2)

0

10

20

30

40

50

Update Num

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Policy Churn (KL)



π

0

10

20

30

40

50

Update Num

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

Policy Value Deviation



π

Q

(1e-2)

(b) Hopper-v4

0

5

10

15

20

Timestep (5

e

4)

0

1000

2000

3000

4000

5000

SAC

SAC w/ VCR (

λQ

=0.1)

SAC w/ VCR (

λQ

=1.0)

SAC w/ PCR (

λπ

=1

e

-4)

SAC w/ PCR (

λπ

=5

e

-4)

0

10

20

30

40

50

Update Num

40

60

80

100

120

Abs. Value Churn



|

Qsa

| (1e-2)

0

10

20

30

40

50

Update Num

-2

-1

0

1

2

Value Churn



Qsa

(1e-2)

0

10

20

30

40

50

Update Num

0

1

2

3

4

Policy Churn (KL)



π

0

10

20

30

40

50

Update Num

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Policy Value Deviation



π

Q

(1e-2)

(c) Walker2d-v4

0

5

10

15

20

Timestep (5

e

4)

1000

2000

3000

4000

5000

SAC

SAC w/ VCR (

λQ

=0.5)

SAC w/ VCR (

λQ

=1.0)

SAC w/ PCR (

λπ

=1

e

-3)

SAC w/ PCR (

λπ

=5

e

-4)

0

10

20

30

40

50

Update Num

10

20

30

40

50

60

70

80

90

Abs. Value Churn



|

Qsa

| (1e-2)

0

10

20

30

40

50

Update Num

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Value Churn



Qsa

(1e-2)

0

10

20

30

40

50

Update Num

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Policy Churn (KL)



π

0

10

20

30

40

50

Update Num

0

1

2

3

4

Policy Value Deviation



π

Q

(1e-2)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

2000

4000

6000

8000

10000

HalfCheetah-v4

TD3

TD3 w/ VCR (

λQ

=0.1)

TD3 w/ VCR (

λQ

=0.5)

TD3 w/ VCR (

λQ

=1.0)

TD3 w/ VCR (

λQ

=5.0)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

0

500

1000

1500

2000

2500

3000

3500

Hopper-v4

TD3

TD3 w/ VCR (

λQ

=0.1)

TD3 w/ VCR (

λQ

=0.5)

TD3 w/ VCR (

λQ

=1.0)

TD3 w/ VCR (

λQ

=5.0)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

0

1000

2000

3000

4000

Walker2d-v4

TD3

TD3 w/ VCR (

λQ

=0.1)

TD3 w/ VCR (

λQ

=0.5)

TD3 w/ VCR (

λQ

=1.0)

TD3 w/ VCR (

λQ

=5.0)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

1000

1500

2000

2500

3000

Ant-v4

TD3

TD3 w/ VCR (

λQ

=0.1)

TD3 w/ VCR (

λQ

=0.5)

TD3 w/ VCR (

λQ

=1.0)

TD3 w/ VCR (

λQ

=5.0)

(a) Value Churn Reduction (VCR)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

2000

4000

6000

8000

10000

HalfCheetah-v4

TD3

TD3 w/ PCR (

λπ

=0.1)

TD3 w/ PCR (

λπ

=0.5)

TD3 w/ PCR (

λπ

=1.0)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

0

500

1000

1500

2000

2500

3000

3500

Hopper-v4

TD3

TD3 w/ PCR (

λπ

=0.1)

TD3 w/ PCR (

λπ

=0.5)

TD3 w/ PCR (

λπ

=1.0)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

0

1000

2000

3000

4000

Walker2d-v4

TD3

TD3 w/ PCR (

λπ

=0.1)

TD3 w/ PCR (

λπ

=0.5)

TD3 w/ PCR (

λπ

=1.0)

TD3 w/ PCR (

λπ

=10)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

1000

1500

2000

2500

3000

3500

Ant-v4

TD3

TD3 w/ PCR (

λπ

=1.0)

TD3 w/ PCR (

λπ

=10)

TD3 w/ PCR (

λπ

=20)

TD3 w/ PCR (

λπ

=50)

(b) Policy Churn Reduction (PCR)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

2000

4000

6000

8000

10000

HalfCheetah-v4

TD3

TD3 w/ DCR (

λQ

=0.05,

λπ

=0.05)

TD3 w/ DCR (

λQ

=0.1,

λπ

=0.1)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

0

500

1000

1500

2000

2500

3000

3500

Hopper-v4

TD3

TD3 w/ DCR (

λQ

=0.05,

λπ

=0.05)

TD3 w/ DCR (

λQ

=0.1,

λπ

=0.1)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

0

500

1000

1500

2000

2500

3000

3500

4000

Walker2d-v4

TD3

TD3 w/ DCR (

λQ

=0.1,

λπ

=0.5)

TD3 w/ DCR (

λQ

=0.5,

λπ

=1.0)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

1000

1500

2000

2500

3000

3500

Ant-v4

TD3

TD3 w/ DCR (

λQ

=0.1,

λπ

=10)

TD3 w/ DCR (

λQ

=0.5,

λπ

=20)

(c) Double Churn Reduction (DCR = VCR + PCR)

̂

̂

̂

̂

̂

̂

̂

̂

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

2000

4000

6000

8000

10000

HalfCheetah-v4

SAC

SAC w/ VCR (

λQ

=0.1)

SAC w/ VCR (

λQ

=0.5)

SAC w/ VCR (

λQ

=1.0)

SAC w/ VCR (

λQ

=0.05)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

500

1000

1500

2000

2500

3000

Hopper-v4

SAC

SAC w/ VCR (

λQ

=0.1)

SAC w/ VCR (

λQ

=0.5)

SAC w/ VCR (

λQ

=1.0)

SAC w/ VCR (

λQ

=0.05)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

0

1000

2000

3000

4000

5000

Walker2d-v4

SAC

SAC w/ VCR (

λQ

=0.1)

SAC w/ VCR (

λQ

=0.5)

SAC w/ VCR (

λQ

=1.0)

SAC w/ VCR (

λQ

=0.05)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

1000

2000

3000

4000

5000

Ant-v4

SAC

SAC w/ VCR (

λQ

=0.1)

SAC w/ VCR (

λQ

=0.5)

SAC w/ VCR (

λQ

=1.0)

SAC w/ VCR (

λQ

=0.05)

(a) Value Churn Reduction (VCR)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

0

2000

4000

6000

8000

10000

HalfCheetah-v4

SAC

SAC w/ PCR (

λπ

=1

e

-4)

SAC w/ PCR (

λπ

=5

e

-4)

SAC w/ PCR (

λπ

=1

e

-3)

SAC w/ PCR (

λπ

=5

e

-3)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

500

1000

1500

2000

2500

3000

3500

Hopper-v4

SAC

SAC w/ PCR (

λπ

=1

e

-4)

SAC w/ PCR (

λπ

=5

e

-4)

SAC w/ PCR (

λπ

=1

e

-3)

SAC w/ PCR (

λπ

=5

e

-3)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

0

1000

2000

3000

4000

Walker2d-v4

SAC

SAC w/ PCR (

λπ

=1

e

-4)

SAC w/ PCR (

λπ

=5

e

-4)

SAC w/ PCR (

λπ

=1

e

-3)

SAC w/ PCR (

λπ

=5

e

-3)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

1000

2000

3000

4000

5000

Ant-v4

SAC

SAC w/ PCR (

λπ

=1

e

-4)

SAC w/ PCR (

λπ

=5

e

-4)

SAC w/ PCR (

λπ

=1

e

-3)

SAC w/ PCR (

λπ

=5

e

-3)

(b) Policy Churn Reduction (PCR)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

2000

4000

6000

8000

10000

HalfCheetah-v4

SAC

SAC w/ DCR (

λQ

=0.1,

λπ

=5

e

-4)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

500

1000

1500

2000

2500

3000

Hopper-v4

SAC

SAC w/ DCR (

λQ

=0.1,

λπ

=1

e

-4)

SAC w/ DCR (

λQ

=0.5,

λπ

=5

e

-4)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

0

1000

2000

3000

4000

Walker2d-v4

SAC

SAC w/ DCR (

λQ

=0.1,

λπ

=1

e

-4)

SAC w/ DCR (

λQ

=0.5,

λπ

=5

e

-4)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

1000

2000

3000

4000

5000

Ant-v4

SAC

SAC w/ DCR (

λQ

=0.1,

λπ

=5

e

-4)

SAC w/ DCR (

λQ

=0.5,

λπ

=1

e

-3)

(c) Double Churn Reduction (DCR = VCR + PCR)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

1000

1500

2000

2500

3000

3500

Ant-v4

TD3

TD3 w/ VCR (

λQ

=0.5)

TD3 w/ PCR (

λπ

=20)

TD3 w/ DCR (

λQ

=0.5,

λπ

=20)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

0

500

1000

1500

2000

2500

3000

3500

Hopper-v4

TD3

TD3 w/ VCR (

λQ

=0.1)

TD3 w/ PCR (

λπ

=0.1)

TD3 w/ DCR (

λQ

=0.05,

λπ

=0.05)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

0

1000

2000

3000

4000

Walker2d-v4

TD3

TD3 w/ VCR (

λQ

=0.5)

TD3 w/ PCR (

λπ

=1.0)

TD3 w/ PCR (

λQ

=0.5,

λπ

=1.0)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

2000

4000

6000

8000

10000

HalfCheetah-v4

TD3

TD3 w/ VCR (

λQ

=0.1)

TD3 w/ PCR (

λπ

=0.1)

TD3 w/ DCR (

λQ

=0.1,

λπ

=0.1)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

1000

2000

3000

4000

5000

Ant-v4

SAC

SAC w/ VCR (

λQ

=0.5)

SAC w/ PCR (

λπ

=1

e

-3)

SAC w/ DCR (

λQ

=0.5,

λπ

=1

e

-3)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

500

1000

1500

2000

2500

3000

3500

Hopper-v4

SAC

SAC w/ VCR (

λQ

=0.5)

SAC w/ PCR (

λπ

=1

e

-3)

SAC w/ DCR (

λQ

=0.5,

λπ

=5

e

-4)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

0

1000

2000

3000

4000

5000

Walker2d-v4

SAC

SAC w/ VCR (

λQ

=0.1)

SAC w/ PCR (

λπ

=1

e

-4)

SAC w/ DCR (

λQ

=0.5,

λπ

=5

e

-4)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

2000

4000

6000

8000

10000

HalfCheetah-v4

SAC

SAC w/ VCR (

λQ

=0.1)

SAC w/ PCR (

λπ

=5

e

-4)

SAC w/ DCR (

λQ

=0.1,

λπ

=5

e

-4)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

2000

4000

6000

8000

10000

HalfCheetah-v4

TD3

CHAIN TD3 (

λπ

=0.1)

CHAIN TD3 (Auto)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

0

500

1000

1500

2000

2500

3000

3500

Hopper-v4

TD3

CHAIN TD3 (

λπ

=0.1)

CHAIN TD3 (Auto)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

0

1000

2000

3000

4000

Walker2d-v4

TD3

CHAIN TD3 (

λπ

=1.0)

CHAIN TD3 (Auto)

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Timestep (5

e

4)

1000

1500

2000

2500

3000

3500

Ant-v4

TD3

CHAIN TD3 (

λπ

=20)

CHAIN TD3 (Auto)

The Effect of Automatic Adjustment of λ π Figure 16 shows the evaluation of the automatic adjustment method of λ π for CHAIN TD3 in four MuJoCo tasks. We can observe that CHAIN (Auto) achieves comparable performance with manually selected coefficients and improves TD3, similarly to the conclusions we found in Figure 4 and 5.

0.0

0.1

0.2

TD3

TD3 w/ VCR

TD3 w/ PCR

TD3 w/ DCR

Median

0.0

0.1

0.2

0.3

IQM

0.0

0.1

0.2

Mean

0.8

0.9

1.0

Optimality Gap

Normalized Score

(a) Final scores in Ant

0.00

0.08

0.16

TD3

TD3 w/ VCR

TD3 w/ PCR

TD3 w/ DCR

Median

0.0

0.1

0.2

IQM

0.00

0.08

0.16

Mean

0.80

0.88

0.96

1.04

Optimality Gap

Normalized Score

(b) AUC scores in Ant

0.08

0.16

0.24

0.32

TD3

TD3 w/ VCR

TD3 w/ PCR

TD3 w/ DCR

Median

0.06

0.12

0.18

0.24

IQM

0.08

0.16

0.24

0.32

Mean

0.72

0.80

0.88

Optimality Gap

Normalized Score

(c) Final scores over all MuJoCo tasks

0.08

0.16

0.24

0.32

TD3

TD3 w/ VCR

TD3 w/ PCR

TD3 w/ DCR

Median

0.08

0.16

0.24

IQM

0.0

0.1

0.2

0.3

Mean

0.7

0.8

0.9

1.0

Optimality Gap

Normalized Score

(d) AUC scores over all MuJoCo tasks

0.4

0.6

0.8

SAC

SAC w/ VCR

SAC w/ PCR

SAC w/ DCR

Median

0.4

0.6

0.8

1.0

IQM

0.4

0.6

0.8

Mean

0.2

0.4

0.6

Optimality Gap

Normalized Score

(a) Final scores in Ant

0.25

0.50

0.75

1.00

SAC

SAC w/ VCR

SAC w/ PCR

SAC w/ DCR

Median

0.3

0.6

0.9

IQM

0.25

0.50

0.75

1.00

Mean

0.25

0.50

0.75

Optimality Gap

Normalized Score

(b) AUC scores in Ant

0.3

0.4

0.5

SAC

SAC w/ VCR

SAC w/ PCR

SAC w/ DCR

Median

0.30

0.45

IQM

0.3

0.4

0.5

Mean

0.5

0.6

0.7

Optimality Gap

Normalized Score

(c) Final scores over all MuJoCo tasks

0.3

0.4

0.5

0.6

SAC

SAC w/ VCR

SAC w/ PCR

SAC w/ DCR

Median

0.3

0.4

0.5

0.6

IQM

0.3

0.4

0.5

Mean

0.5

0.6

0.7

Optimality Gap

Normalized Score

(d) AUC scores over all MuJoCo tasks

D.4 More Results of CHAIN for IQL with Sequential Training

As mentioned in Section 6.3, the policy network of IQL has no impact on the training of the value networks, since the value networks (i.e., Q and V ) are trained purely based on in-sample data without accessing a ′ = π ϕ ( s ′ ) . Thus, although the policy and value networks of IQL still have churns, the chain effect of churn does not apply in this case.

The default implementation of IQL follows the fashion of iterative training between the policy network and the value network(s). Since the training of the value networks of IQL is independent of the policy, one natural idea is to first fully train the value networks for a sufficient budget and then train the policy value with the well-trained and frozen value networks. We call this actor-trainedagainst-final-frozen-critic fashion as sequential training of IQL.

Weslightly modified the training process of the CORL implementation of IQL to realize the sequential training of IQL: (1) First train the value network and Q network for 1M steps; (2) Then train the policy network for 1M steps with the value network and Q network frozen; (3) We do not modify any other implementation detail and use the same hyperparameters; (4) We check the learning curves of the policy network and the final scores. We call this variation IQL (sequential) . The total number of gradient step is the same as the default IQL implementation where the critic and actor are trained iteratively.

We report the final scores of IQL (sequential) with means and standard errors over 12 seeds in Table 8. The results show that IQL (sequential) performs worse than IQL in 5 of 6 tasks. The difference between IQL (sequential) and IQL can be fully attributed to the difference in the training dynamics, mainly on the policy network. This also means exposing the actor network to the training process of the Q network is helpful compared to training the actor based on a sufficiently trained and frozen Q network. To provide some possible explanation, we guess that this is because the final Q network is a product of accumulated value churns, and the difference in the training dynamics of the policy network of IQL results in a further difference in the policy output on out-of-sample states. However, it is somewhat tricky to explain the difference between IQL (sequential) and IQL. The dynamics is beyond the scope of the chain effect of churn mechanism studied in our work.

D.5 More Results for Scaling PPO with CHAIN

We take PPO and MuJoCo tasks as the exemplary setting and widen both the policy and value networks by a scale-up ratio within { 2 , 4 , 8 , 16 } . Note that the default network architecture (i.e., when the scale-up ratio equals one) for both the policy and value networks is a two-layer MLP with 256 neurons for each layer, followed by an output layer.

Inspired by the prior study [Obando-Ceron et al., 2024], in addition to directly scaling PPO up ('direct'), we use two variants that use different learning rate settings for comparison: (1) 'linear' means using a decreased learning rate as lr / scale-up ratio , and (2) 'sqrt' means using lr / sqrt(scale-up ratio) . The results of scaling PPO with different learning rate settings are shown in Figure 19 by different colors.

As expected, we observed that the performance of PPO degrades as the increase of the scale-up ratio severely. We found using a decreased learning rate with 'linear' or 'sqrt' alleviates the degradation of PPO scaling to some degree.

The evaluation results of the effect of CHAIN in this scaling setting are shown in Figure 7, Figure ?? and Table 3 as discussed in the main body of this paper.

M)

atio

8

16

Performance (score)

2500

2000

1500

1000

500

1

Humanoid-v4 (2M)

2

1

2

4

8

16

Scale Up Ratio

2000

2500

3000

3500

4000

Performance (score)

Walker2d-v4 (2M)

direct

sqrt_lr

linear_lr

1

2

4

8

16

Scale Up Ratio

0

500

1000

1500

2000

2500

Performance (score)

Ant-v4 (2M)

1

2

4

8

16

Scale Up Ratio

500

1000

1500

2000

2500

Performance (score)

Humanoid-v4 (2M)

4

8

16

1

2

4

8

16

Scale Up Ratio

1800

1900

2000

2100

2200

2300

2400

Performance (score)

Hopper-v4 (2M)

1

2

4

8

16

Scale Up Ratio

1400

1600

1800

2000

2200

Performance (score)

HalfCheetah-v4 (2M)

Scale Up Ratio

1

2

4

8

16

Scale Up Ratio

0

500

1000

1500

2000

2500

3000

Performance (score)

Ant-v4 (2M)

PPO

PPO (w/ sqrt_lr)

CHAIN PPO

CHAIN PPO (w/ sqrt_lr)

1

2

4

8

16

Scale Up Ratio

500

1000

1500

2000

2500

3000

3500

Performance (score)

Humanoid-v4 (2M)

1

2

4

8

16

Scale Up Ratio

2000

2500

3000

3500

4000

Performance (score)

Walker2d-v4 (2M)

(a) PPO Scaling by

Widening

1

2

4

8

16

Scale Up Ratio

500

1000

1500

2000

2500

Performance (score)

Ant-v4 (2M)

PPO

PPO (w/ sqrt_lr)

CHAIN PPO

CHAIN PPO (w/ sqrt_lr)

1

2

4

8

16

Scale Up Ratio

500

1000

1500

2000

2500

Performance (score)

Humanoid-v4 (2M)

1

2

4

8

16

Scale Up Ratio

1500

2000

2500

3000

3500

Performance (score)

Walker2d-v4 (2M)

(b) PPO Scaling by

Deepening

D.6 More Results for CHAIN v.s. Slowing Down Learning

Since churn accompanies each time mini-batch training performed for the networks, a natural question is: whether slowing down learning can alleviate the issue of churn. In principle, using smaller learning rates or target network replacement rates should lead to less churn. This is because churn is positively related to the parameter update amount (as shown by the NTK expressions in Eq. 2) and a slower target network also slows the churn that occurs instantly in each training more when computing the target value to fit for the critic-network.

Empirically, we ran DDQN in Asterix and Freeway and TD3 in Walker2d and Ant, with different learning rates and target network replacement rates. The results are summarized in Table 9. We can observe that either reducing learning rate or target network replacement rate often leads to worse

Performance (score)

2400

2300

2200

2100

2000

1900

1800

1

2

Hopper-v4 (2M)

4

Scale Up Ratio

8

performance, especially for TD3. To some extent, this also matches the common knowledge in the RL community.

This indicates that the issue of churn cannot be addressed by reducing learning rate or target network replacement rate (which usually slows down the learning process). Churn is a 'by-product' of the training of DRL agents and should be addressed separately.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims in the abstract and introduction summarize our contributions and are supported by our formal analysis in Section 4 and our experiment results in Section 6.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations in Appendix A.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide sufficient experiment and implementation details in Section 6 and Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Our code can be found at https://github.com/bluecontra/CHAIN .

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide necessary experimental details in Section 6 and Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The results in our paper are accompanied by error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the details of compute resources in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This work conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper presents work whose goal is to advance the field of Deep Reinforcement Learning (DRL). No specific real-world application is concerned. Our study on the churn phenomenon may help us better understand and control the behaviours of DRL agents, which can have a positive impact in general on the applicability and safety of DRL techniques.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All the environments, data and codes of baseline methods used in this paper are publicly available on Github. We cited the original papers and provided the URLs to the assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide the necessary details for implementing our proposed method in this paper. We will provide a README document alongside the code to release after the review process.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies a phenomenon called churn, which refers to that the outputs of a network after updates could change to unexpected values for input data not included in the training batch.\nSpecifically, it studies the value churn, the policy churn, and the interplay between them in deep reinforcement learning (DRL) setting.\nThen a simple regularization method is propose to reduce the the value and policy churn. Empirical results demonstrate the effectiveness of the proposed solution across many DRL settings, such as value-based methods, policy gradients methods, and offline RL.\n\nSTRENGTHS:\n- Except some minor issues, the paper is generally well-written and easy to follow.\n- The proposed method is simple and effective, verified in various RL settings.\n- Besides improving the final performance, direct evidences are also provided to show that the proposed method indeed can reduce churns.\n\nWEAKNESSES:\n## Originality\n\nAlthough this work studies a relatively new concept (churn), the investigations of the root causes of this phenomenon, the proposed method, and the idea behind are not novel.\n  - The churn phenomenon is strongly related to interference and generalization of neural networks. And using NTK to measure the interference and generalization is already proposed by previous works for DRL, such as [1-4]. These works should be included in the related works and discussed. Out of DRL, similar phenomena are also studied under supervised learning setting, such as [5-7].\n  - I like the simplicity and effectiveness of the proposed algorithm (CHAIN), but it is really similar to MeDQN(R)[8] although the latter work focuses on reducing forgetting instead. This greatly damages the novelty of this work unfortunately.\n\n\n## Quality\n\nThe empirical and theoretical results of this work can be further improved.\n  - Equation 1 is inappropriate as the definition of churn. An absolute operation or square operation should be applied. Otherwise, positive and negative values may cancel out. The issue exists for Equation 4 as well. This is a serious issue since changing Equation 1 would also results in changing the latter derivations in the paper. For example, Equation 5 may no longer hold.\n  - In Equation 3, although using $B_{train} = \\\\{s,a\\\\}$ and $B_{ref} = \\\\{\\bar{s}, \\bar{a}\\\\}$ help with notation clarity. However, it may lead to misleading derivations. For example, the cancel-out effect is no longer revealed (see previous point about Equation 1). Given this, I would suggest to consider the general case of using more samples in $B_{train}$ and $B_{ref}$.\n  - All experiments use only 6 random seeds which are not enough. I would suggest at least 10 seeds.\n  - The theoretical results are interesting and useful to help understand and support the intuitions behind the problem. However, I don't see how they could help with the proposed algorithm. In other words, the connection between theoretical derivations and the proposed method is too weak.\n\n## Clarity\n\nThere are some minor issues.\n  - What is the difference between $Q\\_{\\theta}$ and $\\tilde{Q}\\_{\\theta}$ ($\\pi\\_{\\phi}$ and $\\tilde{\\pi}\\_{\\phi}$)? I understand that $Q_{\\theta}$ is an approximation of $q^{\\pi}$ (Section 2.2) where $\\theta$ are the parameters of the Q-network. But what is $\\tilde{Q}\\_{\\theta}$ in Section 3.1 and Figure 1 exactly? Isn't it also an approximation of $q^{\\pi}$? In fact, I would suggest to remove $\\tilde{Q}\\_{\\theta}$ and $\\tilde{\\pi}\\_{\\phi}$ to simplify notations, as they seem to be redundant (but I might be wrong).\n  - Missing one right parenthesis in the definition of $\\delta(s,a)$ in Line 110.\n  - Typo in Line 322: policy churn reduction (VCR) --> value churn reduction (VCR).\n\n\n1. Liu, Vincent, et al. \"Towards a practical measure of interference for reinforcement learning.\" arXiv preprint arXiv:2007.03807 (2020).\n2. Liu, Vincent, et al. \"Measuring and mitigating interference in reinforcement learning.\" Conference on Lifelong Learning Agents. PMLR, 2023.\n3. Joshua Achiam, Ethan Knight, and Pieter Abbeel. Towards characterizing divergence in deep Q-learning. arXiv:1903.08894, 2019.\n4. Emmanuel Bengio, Joelle Pineau, and Doina Precup. Interference and generalization in temporal difference learning. arXiv preprint arXiv:2003.06350, 2020.\n5. Fort, Stanislav, et al. \"Stiffness: A new perspective on generalization in neural networks.\" arXiv preprint arXiv:1901.09491 (2019).\n6. He, Hangfeng, and Weijie Su. \"The Local Elasticity of Neural Networks.\" International Conference on Learning Representations.\n7. Lan, Qingfeng, and A. Rupam Mahmood. \"Elephant neural networks: Born to be a continual learner.\" arXiv preprint arXiv:2310.01365 (2023).\n8. Lan, Qingfeng, et al. \"Memory-efficient Reinforcement Learning with Value-based Knowledge Consolidation.\" Transactions on Machine Learning Research.\n\nQUESTIONS:\nHow does policy/value churn related to catastrophic forgetting? For example, Figure 2 in MeDQN paper[8] shows that a change of greedy action could result in forgetting. Will reducing forgetting reduce churn? Or vice versa?\n\nLIMITATIONS:\nYes, the limitations are discussed in the work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors focus on improving the current state of deep reinforcement learning by addressing the churn effect in deep neural network training. Churn effect in deep reinforcement learning is the phenomenon where output predictions for data outside training dataset can change during training updates. This can lead to instability, suboptimality and even collapse. Previous works in addressing churn are limited in scope and do not provide a method to counter churn effect. In this work, the authors perform a detailed exploration of churn effect in RL and propose an algorithm to counter this issue. They characterize churn in a view of Generalized Policy Iteration with function approximation and discover a chain effect of churn. This chain effect leads to a cycle that compound and bias the learning dynamics throughout the iteration. The authors study the effect of churn across different setups and propose CHAIN algorithm, which can be easily plugged into most existing DRL algorithms. Experiments show that their method is effective in both reducing churn and improving learning performance across online and offline, value-based and policy-based RL settings.\n\nSTRENGTHS:\nThe problem is well motivated, the authors provide detailed explanations of what the problem is, why its important, what has been done previously, and a detailed explanation of their methodology.\n\nWEAKNESSES:\n1.\tIn the introduction, please provide a short intuition behind the CHAIN algorithm, how is it achieving the explicit control of churn. Perhaps mention that by modifying the original loss function, CHAIN can achieve this.\n2.\tExpand on the figure captions. This applies to all figures 1-6. \n3.\tFor figure 1, what is meant by the policy churn and value churn arrows? Are there any updates going on in between? In the description, the authors mentioned “explicit training”, please elaborate what is meant by this.\n4.\tFor figure 2, expand the caption for 2b, in the figure 2a there is an arrow without direction, please double check if this is accurate, if yes mention in the caption why that is the case\n5.\tFor figure 3-6, mention the key takeaways from the graphs in figure caption\n6.\tIn table 1, CHAIN seems to perform worse for AM-large-diverse-v2 dataset, any explanation/intuition as to why that may be the case?\n7.\tFigure 3 appears after figure 4, please correct this.\n\nQUESTIONS:\nPlease see weakness section\n\nLIMITATIONS:\nYes, the authors mention limitations in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDeep RL optimisation exhibits many instabilities and performance collapses. Schaul et al. [2022] discuss a pathology termed the \"policy churn\", where even a single update to the value network frequently changes the optimal action for a huge fraction of all states (most of which were not present in the training batch). Following in these steps, the current paper characterises churn through the lens of generalised policy iteration, and proposes a simple regularisation loss that can be intuitively described as restricting the actor and critic predictions from changing \"too much\" on other states.\n\nSTRENGTHS:\nThe authors argue convincingly that churn effects in actor and critic optimisation can interact amplify each other, in what they call the \"chain effect\", and propose a surprisingly simple regularisation loss to weaken this effect. As a result, the churn measurably goes down, which appears to correlate with better learning performance.\nAs there are few papers on mitigating churn, I think these findings are useful and can help inform further research.\n\nWEAKNESSES:\n# Writing\nI have some concerns about the writing. The topic is important, and relatively new, so good communication would be valuable.\n\nThe paper is quite well written until page 4, but later, it gets hard to follow. \nFor example, the language starts to get more vague, e.g. line `176` -\n> One thing to note is the two types of deviation are derived in an **exchange manner** between the value and policy churn,\n\nI found Section `3.3` particularly hard to follow and had to reread it many times. It introduces a lot of new symbols in one go, uses them in 1/3rd of the page, and then moves on to the next topic. The section's conclusion is that churn and parameter update bias amplify each other as the optimization goes on (something which the reader is already informally aware of since early on in the paper).\n\nAfterwards, the main method (CHAIN) is introduced as late as *page 7*, following which the paper appears quite rushed.\n\nI think the paper could benefit from:\n- moving the NTK discussion (which is never used afterwards) and everything theoretical after that to the appendix. The math may be saying the right things, but I did not find it *useful* in this context.\n- focusing on describing CHAIN and its experiments in more detail.\n\n`326`: You mention Fig 5, but I think you mean Fig 6\n\n`328`: I'm not sure what the following bolded part means. Does it mean that the target critic network slows down credit assignment / weight updates, and therefore reduces churn? It would be helpful to be more verbose in the text right here. \n> We hypothesize that this is because policy interacts with the environment directly **and the target critic network also helps to cap the value churn**. \n\n\n# Experiments\n\nThe churn reduction looks quite noticeable and the MinAtar evaluations also show a notable performance boost. How much of this observed churn reduction can be explained by simply \"slowing down\" the parameters' changes? \n\nI think this is quite important to test, one could also simply train both actor and critic with:\n1. slightly reduced learning rates\n2. different target network momentums\n... and observe if the churn also reduces due to those things (and whether the agent performance also correspondingly improves). I believe such an experiment would give clearer insights, and possibly strengthen the paper.\n\n\nLine `624`: \n>  Between TD3 and SAC, CHAIN-PCR works better for TD3 rather than SAC. We hypothesize that this is because KL-based PCR used for SAC poses more challenges in code implementation.\n\nThat is concerning, because it sounds like a bug in your code. Not affecting the entire paper, but it does make it hard to believe the SAC results.\n\nQUESTIONS:\n**IQL**: Line `338` says you apply CHAIN to IQL --- could you explain how? as far as I can tell, IQL does not need to train an actor network in tandem with a critic. It's likely that I did not fully understand this part.\n\nLIMITATIONS:\nThe authors do mention the limitations of the work, but the paper could be greatly improved by reducing the role of the mathematical exposition that takes up the majority of the paper, and giving more space to the experiments.\n\nThis is an important area of work that can benefit from new empirical findings, and I am quite willing to improve my rating if my concerns are addressed.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focues on the training instability from the non-stationary nature of DRL, whose pheononmena are unexpected shift in policy and value network outputs (policy churn and value churn). To mitigate policy and value churn, the proposed CHAIN algorithm add penalize in the policy and value updates.\n\nSTRENGTHS:\nCHAIN is a versatile solution that can be integrated into different backbone algorithms.\n\nWEAKNESSES:\n* Extra introduced hyperparameters and no-trivial hyperparameters setting. \n* No statistically significant improvements in many tasks, even equipped with good hyperparameters.\n\nQUESTIONS:\n1. CHAIN introuduces two extra coefficients, $\\lambda_Q$ and $\\lambda_\\pi$ to control the degree of regularization. I find that for different benchmark suites or tasks or backbone algorithms, are set differently. So I have the following questions:     \n\n   What is the insight behind the chosen hyperparameters for different settings? If the potential readers want to apply CHAIN to other settings, how can they set the two hyperparameters? Actually, I find some explanations on this in Appendix C.2, but it requires making [policy loss]/[regularization loss] within 0.1 to 0.01, thus it needs to first try and then observe the ratio and then turn, maybe need several runs, whose costs could not be ignored.  Introducing extra hyperparameters and a simple one set of them could not suffice to get satisfying results for varying settings is not encouraged, as actually, it may output worse performance than the backbone algorithm if setting wrong hyperparameters. \n\n2. How many seeds do you use for the experiments in your main paper? For example, Figure 6. The CHAIN SAC in Figure 6 does not show a statistically significant improvement to SAC in Hopper-v4, Walker2d-v4, and HalfCheetah-v4, which may bring some doubts on the effectiveness of CHAIN, especially considering that an unsuitable hyperparameters set may provide even worse results of CHAIN SAC.\n\n\n\n3. As the paper focuses on mitigating the training instability of DRL and the non-stationary properties, some other methods may need to be considered to compare to or discuss, such as AlgaeDice [1], etc. \n\n\n\n4. Further, in online RL, over-conservative may hinder performance, so maybe the authors need to discuss well: whether the CHAIN would negatively influence the exploration in online RL and then hinder the asymptotic performance.\n\n5. And, another style of work, for example, [2],  in online RL actually encourages the network neurons to awaken, thus preventing the local optimal. Could you provide some explanations on whether CHAIN would deteriorate the local optimality issue?\n\n[1] Nachum, O., Dai, B., Kostrikov, I., Chow, Y., Li, L., & Schuurmans, D. (2019). Algaedice: Policy gradient from arbitrary experience. *arXiv preprint arXiv:1912.02074*.\n\n[2] Xu, Guowei, et al. \"Drm: Mastering visual reinforcement learning through dormant ratio minimization.\" *arXiv preprint arXiv:2310.19668* (2023).\n\nLIMITATIONS:\nThe authors have acknowledged the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-14"}