Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churn
Authors: Hongyao Tang
Glen Berseth
Venue: NeurIPS 2024 poster (2024)
Keywords: Reinforcement Learning
Deep Learning
Regularization and Optimization
Primary area: reinforcement_learning
TL;DR: Improving deep reinforcement learning by reducing undesired changes to out of batch data.

Abstract: Deep neural networks provide Reinforcement Learning (RL) powerful function approximators to address large-scale decision-making problems. However, these approximators introduce challenges due to the non-stationary nature of RL training. One source of the challenges in RL is that output predictions can churn, leading to uncontrolled changes after each batch update for states not included in the batch. Although such a churn phenomenon exists in each step of network training, it remains under-explored on how churn occurs and impacts RL. In this work, we start by characterizing churn in a view of Generalized Policy Iteration with function approximation, and we discover a chain effect of churn that leads to a cycle where the churns in value estimation and policy improvement compound and bias the learning dynamics throughout the iteration. Further, we concretize the study and focus on the learning issues caused by the chain effect in different settings, including greedy action deviation in value-based methods, trust region violation in proximal policy optimization, and dual bias of policy value in actor-critic methods. We then propose a method to reduce the chain effect across different settings, called Churn Approximated ReductIoN (CHAIN), which can be easily plugged into most existing DRL algorithms. Our experiments demonstrate the effectiveness of our method in both reducing churn and improving learning performance across online and offline, value-based and policy-based RL settings.

Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churn
Hongyao Tang
Glen Berseth
Mila Québec Al Institute Université de Montréal
Mila Québec AI Institute
Université de Montréal
tang hww.yoonnii la quebec
glen aaareet@@mila quebec
Abstract
Deep neural networks provide Reinforcement Learning (RL) powerful function approximators to address large-scale decision-making problems. However, these approximators introduce challenges due to the non-stationary nature of RI training. One source of the challenges in RL is that output predictions can churn, leading to uncontrolled changes after each batch update for states not included in the batch. Although such a churn phenomenon exists in each step of network training, how churn occurs and impacts RL remains under-explored. In this work, we start by characterizing churn view of Generalized Policy Iteration with function approximation, and we discover chain effect of churn that leads to a cycle where the churns in value estimation and policy improvement compound and bias the learning dynamics throughout the iteration. Further, we concretize the study and focus on the learning issues caused by the chain effect in different settings, including greedy action deviation in value-based methods, trust region violation in proximal policy optimization, and dual bias policy value in actor-critic methods. We then propose method to reduce the chain effect across different settings, called Churn Approximated ReductIoN (CHAIN), which can be easily plugged into most existing DRL algorithms. Our experiments demonstrate the effectiveness of our method in both reducing churn and improving learning performance across online and offline, value-based and policy-basee RL settings, well scaling setting.
Introduction
One fundamental recipe for the success of Deep Reinforcement Learning (DRL) is powerful approxi- mation and generalization provided by deep neural networks, which augments the ability of RL with tabular linear approximation to large state spaces. However, the other side of this benefit is less control over the function dynamics. Network outputs can change indirectly to unexpected values after any random batch update for input data included in the batch, called churn in this paper. This change is particularly problematic for an RL agent due to its non-stationary nature, which can exacerbate instability, suboptimality, and even collapse. Therefore, control these undesired dynamics to address learning issues and improve performance.
important to understand and
Consistent efforts have been devoted by the RL community gain a better understanding of the learning dynamics from different perspectives [Achiam et al.) 2019 Kumar et al.) 2022 Liu et al.
2023 Lyle al. 2022b]. . Recently, Schaul et al. [2022] studied novel churn phenomenon in the learning process of typical value-based RL algorithms like DoubleDQN [van Hasselt et al., 2016).
The phenomenon reveals that the greedy actions of about 10% of states in the replay buffer change after a single regular batch update. Such a dramatic churn can persist throughout the learning process of DoubleDQN, causing instabilities.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

In this paper, we aim to take a step further to understand how churn occurs and influences learning in different DRL settings beyond value-based RL, as well as to propose a method to control churn. We start by formally characterizing churn in view of Generalized Policy Iteration (GPI) with function approximation to best cover most DRL settings. The impact of churn is two-fold in this view: the churn in policy improvement (called the policy churn) changes policy outputs on states that are not directly updated, while the churn in value estimation (called the value churn) also changes the action-value landscape, thus altering greedy action and action gradient. We then discover chain effect of churn that exhibits a cycle where the two types of churn compound and bias the learning dynamics throughout the iteration. Further, we move on from the general analysis to concrete DRL settings. We focus on the learning issues caused by the chain effect including greedy action deviation in value-based methods, trust region violation in proximal policy optimization (Schulman al. 2017] and dual bias of policy value in actor-critic methods. The connection between the chain effect of churn and the issues necessitates an explicit control of the churn.
To this end, we propose a method called Churn Approximated ReductIoN (CHAIN) to reduce the chain effect of churn across different DRL settings. The main idea of CHAIN is to reduce the undesirable changes to the outputs of the policy and value networks for states (and actions) outside of the current baich of data for regular DRL training. This reduction is achieved by minimizing the change in target values for a separate batch of data when optimizing the original policy or value learning objective, CHAIN is easy implement and plug in most existing DRL algorithms with only few lines of code In our experiments, we evaluate the efficacy of CHAIN in range of environments, including MinAtar (Young and Tian 2019]. OpenAl MuJoCo [Brockman et al. 2016). , DeepMind Control Suite Tassa et al. 2018) and D4RL. (uu et al. 2020]. The results show that our method can effectively reduce churn and mitigate the learning issues, thus improving sample efficiency final performance across online and offline, value- based and policy-based DRL settings. Moreover, our results also show that our method helps to scale DRL agents up and achieves significantly better learning performance when using wider or deeper networks.
The main contributions of this work are summarized as follows: (1) We study how churn occurs and influences learning from the perspective of GPI with function approximation and present the chain effect of churn. (2) We show how churn results in three learning issues in typical DRL settings. (3) We propose a simple and general method and demonstrate its effectiveness in reducing churn and improving learning performance across various DRL settings and environments.
2
Prior Work
In the past decade, a significant effort has been made to understand the learning issues of DRL agents and propose improvements that make DRL more stable and effective. The early stage of this effort studied bias control for value approximation with deep neural networks introducing many improvement improvements after DQN (Mnih et al. 2015) and DDPG [Lillicrap
al. 2015) address overestimation or underestimation for value-based methods [van Hasselt et al. 2016 Bellemare et al.
2017 Hessel al., [2018] and deep AC methods [Fujimoto et al. a [2018] Haarnoja et al. [2018] Lan et al. 2020 Kuznetsov et al.] 2020 Chen et al. 2021] respectively. Additional works dig deeper to diagnose the learning issues regarding instability and generalization, related the Deadly Triad
in DRL (van Hasselt et al.] 2018 Achiam et al. 2019]. stabilizing effect of target network (Zhang
let al.]] [2021b] et al. [2022] Piché al. 2022), difficulty of experience replay [Schaul et al. 2016
Kumar et |l.]|020000 Ostrovski et al. [2221], J, over-generalization [Ghiassian et al. | 2020
Pan Pan et al.
2021
Yang et al. 2022], representations in DRL [Zhang et al. [2021a] Li et al. 2022 Tang et al.
2022 delusional bias [u et al. 2018 Su et al., 2020). off-policy correction Nachum et al. 2019
Zhang et al.) 2020 Lee et al. 2021], interference [Cobbe et al. 2021 Raileanu and Fergus 2021
Bengio et al. 2020 and architecture choices [Ota et al., 2020).
One notable thread is to understand the learning dynamics of DRL agents with a focus on the non-stationary nature of RL. A prominent phenomenon of representation ability loss is studied
in [Danney al.) 2021 [g] al. 2021 Kumar et al. representations become less useful in later stages of learning, leading to myopic convergence. Further,
2021
2022 Ma et al. 2023], which reveals how
empirical studies in [Nikishin et al. 2022 D' Oro et al. 2023] Sokar et al., 2023 Nauman al.) 2024]
demonstrate that the loss of approximation ability becomes severe and leads to collapse when high
2

---

replay-ratios are adopted for better sample efficiency, while network resets and normalization methods can be simple and effective remedies. This is further identified as plasticity loss in DRL (Lyle et al.) 2022a
Abbas et al. 2023 Dohare et al. 2023 Lyle et al. 2024 Xu et al. 2024
Recently, it has been found that there is dramatic change in the policy distribution where large portion of the greedy actions change after each batch update, called policy churn /Shhul et al. 2022]
Although intuitively related generalization [Bengio al. 2020] and interference [Liu et al. 2023
it presents a lack of understanding of churn's 's effect on the learning behaviors of DRL agents regarding
stability, convergence, exploration, etc. Kapturowski et al. 2023] takes the inspiration and proposes a method to robustify the agent's 's behavior by adding an additional policy head to the value network that fits the e-greedy policy via policy distillation. In this work, we further the study of churn in a more general formal framework, where churn occurs in both value and policy learning. In particular, we focus on the dynamies of churn during the learning process and how it incurs issues in different DRI. algorithms and propose practical method to reduce churn and improve learning performance.
3 Preliminaries
Reinforcement Learning (RL) is formulated within the framework of a Markov Decision Process (MDP) (S, A. P. R. ", Po. T). defined with the state set S, the action set A, the transition function y..
P A - P(S), the reward function R: S x - R. the discounted factor y E [0, 1), the initial state distribution Po and the horizon T. The agent interacts with the MDP by performing actions from its policy a, (T(s)) that defines the mapping from states to actions action distributions. The objective of an RL agent is to optimize its policy to maximize the expected discounted cumulative reward (T) = E, |-== r'rel, where So ~ Po (so), St+1 ~ P (s 8t, az) and = R (st, at). The state-action value function q" defines the expected cumulative discounted reward for all s, e x A and the policy n, i... q= a) = E,, [>: 7'1, 50 = 8, 8,00 = a].
Policy and value functions are approximated with deep neural networks to cope with large and continuous state-action space. Conventionally, 9ºd can be approximated by Qe with parameters typically through minimizing Temporal Difference (TD) loss [Sutton and Barto 1988 i.e., L(0) = Ex,e~D (s, a)² where D is a replay buffer and (s a) a type of TD error. A parameterized policy To with parameters e can be updated by taking the gradient of the objective, i.e., ø'' <- + (T))))
with a step size a. Value-based methods like Deep N-Ntworrk (DQN) [Mnih et al. 2015 trains a Q-network Qo by minimizing [.)) where ö(s, a) = Qa a) (r + 7 max,' Q8- (8' denotes the target network. For policy-based methods, TD3 [Fujimoto et al., 2018 is often used to update deterministic policy with Deterministic Policy Gradient (DPG) theorem (Silver et al. 2014 (TT( = Ea-D VoT (s) QO(K, a) a))== Soft Actor-Critic (SAC) HHarrooja et al. 2018
and A-
learns stochastic policy with the gradient: (TT) Eg~D VV log n (als) + (V. log (s(a
VoQo(s. V s)))a=)((ss)) with noise e and implicit function fø for pe-paaameterizttion.
(f
4 A Chain Effect of Value and Policy Churn
In this section, we present formal study on value and policy churn and their impact on learning. We first introduce an intuitive overview of how churn is involved in DRL (Section (.1)). Then, we propose the definitions of the value and policy churn (Section 4.2). followed by a chain effect that reveals how the churns interplay and bias parameter update (Section 4.3
4.1 Generalized Policy Iteration under Churn
Generalized Policy Iteration (GPD) [Sutton and Barto 1988 is widely used to refer to the general principle of learning in an Evaluation- Improvemmnt iteration manner, which applies to almost all RL methods. In the context of DRL, i.c., with network representation and mini-batch training, the value and policy networks' outputs can have unexpected changes, i.e., the churn, after each mini-batch training for the states not included in the batch. Such churns are neglected in most DRL methods, let alone their influence on the practical learning process. In Figure [ we extend the classic GPI diagram by considering churn to show how it is involved in the learning process intuitively.
In the evaluation process, the parameterized Q-network Qo approximates the value of the current policy via repeated mini-batch training. Under the impact of churn, the Q-network is not likely to
3

---

have output predictions the same as what was updated with explicit mini batch training. For example, let's imagine we have virtual network Qo that only accepts the changes for the states updated by mini-batch training directly and remains unchanged for the others.
Thus, the value churn, denoted by Qo Qo- is im- plicit process that alters the virtual network Qo to the approximation Qe we obtained in practice. Similarly, the policy churn & ~- V occurs in the improvement process, As illustrated in Figure ! the value churn and the pol- icy churn are interwoven in the Evaluation-Impien Improvement process. Usually, we can assume that the churns make non-negligible changes, i.e., Q9 * Qo and T # Tg.. Therefore, we delve into the cause of churn, its impact on learning, and possible remedies to mitigate its negative impact in the following sections.
Evaluation Qo g"
Qo
Policy
Value Churn
Charn
T
greedy(Qs)
Improvement
Figure I: Generalized Policy Iteration (GPI) under the value and policy churn.
4.2 Definition of Value and Policy Churn
deep neural network can have the form fo :X: + Y with parameters The network is optimized for a set of input data Brain {ax²} } with a loss function, leading to a parameter update of 0 0'. Given reference set of input data Brer = { (where Brer O Biaain = 0) and a metric d for the output space, the churn is formally defined as:
C, 0', Birr)
Σ d(So fo),
|
BEBu
Arguably, churn an innate property of neural networks, and is closely related problems like interference (Liu et al. 2020 2023 and catastrophic forgetting [Lan et al. 2023] in different contexts.
In this paper, we focus the churn in Q-value network Qo and policy network TT. We then obtain the definitions the Q-value churn (Cq, w.r.t. -> 8') and the policy churn (C,, w.r.t. & - o', using deterministic policy for demonstration) for an arbitrary sate-action pair 5, Buet follows:
cQ 0', 5, a) Q (5, a) (0 a), (r d', 8) Te (3) (T))).
(1)
Then, the definitions regarding Bier can be generalized to the batch setting by aggregating data in Biet CO,0, 0', Bret) = [hazl B, Icg 3,, a)), Cz d', Bret) Barl Σee But |ca (Ø, o' 8)]. Without loss of generality, we carry out our analysis mainly regarding 8, a for clarity in the following.
(U.I) How the churns Cq. Cx caused by parameter updates First, look into the relationship between the Q-value churn Co. the policy churn C, and the network parameter updates Ae 0' 0, As Øb' Φ.. For Do, Ag. we use typical TD learning and DPG for demonstration: As
Brian Ex, Bruin V PoQu(s, a))u(s,a), and A. Bininl ExeRp (s) a) a)la=ra(s)-
Now we characterize Co and C2 as functions of Ag, Ag with the help of Neural Tangent Kernel (NTK) [Achiam al. 2019). For clarity, we use Brrin {s, and Bie {5, a} and abbreviate Braain, Bier and step size a when context is clear. Concretely,
(0) voQo(s. a) Ao O(1
VoQo(3,a) VoQo(s, (s(s,
ko,3,,0s,a)
cx c= V (3)
A O(IA
(T) (s
Qo (s) (3)==
(8,9)
Eq. [2] shows that the value and policy chum are mainly determined by the kernels of the Q-network kg and the policy network ky. along with the TD eттor and the action gradient. This indicates that chum is determined by both the network's 's property itself and the learning we performed with the network.
4.3 From Single-step Interplay to The Chain Effect of Churn
In addition to the first piece of understanding (U.I) that presents how parameter updates cause the churns, we discuss how the churns affect parameter updates backward with two more pieces of understanding (U.II) and (U.II), finally shedding light on chain effect of chum.

---

(UIII) How Cz deviates action gradient and policy value First, we introduce two types of deviation derived from the value and policy churn: (1) Action Gradient Deviation (D ). the change of action gradient regarding the Q-network for states and actions that are affected by the Q-value churn Co: (2) Policy Value Deviation (Dz), the change of Q-value due to the action change for states that are affected by policy churn C.. Formally, the two types of deviation are:
& (0,0'' 8) = VaQu (3, ()aa)==(()) - VaQo(3, a) (=x(s)
(0,00' 8) = Q Te (8)) - (s, T (8)).
One thing to note is the two types of deviation show the interplay between the value and policy churn, as the value churn derives the deviation in policy (cQ
and the policy churn derives the deviation in value (cz derive, dg)), as denoted by the superscripts. This interplay between the policy and value can be shown better with the expressions below (derivation details in Appendix BB:
derive,
(0,00 = (a 0' (x);;
(0 (')= ~ (VaQo (.))))=))((()))) c+ x¹)
Since the action gradient and policy value play key roles in parameter updates, the deviations in them naturally incur negative impacts on learning.
The discussion thus far is within a single parameter update. Now, we discuss the implications of these single updates towards chain of updates to shed light on the long-term effect of churn.
(U.III) How parameter updates are biased by Cq. Cx and the deviations D& Dz Let us consider a segment of two consecutive updates, denoted by (0 . )- - (0, 6) - (0', g'). The churns occurred during the last update (0 o + (9, 6) participate in the current update (0, Φ) - 00', about to perform. Concretely, the churns affect the following aspects: (1) Q-value estimate, (2) action selection in both TD error and policy objective, and (3) the gradient of network parameters.
From these aspects, we can deduce the difference between the parameter updates under the impact of the value and policy churn (denoted by Åo, A&) and the conventional ones مe, Ag. As result, we can find that the value and policy churn, as well as the deviations derived, introduce biases in the parameter updates. We provide the complete discussion and derivation in Appendix B.2
The analysis on the update segment "- - (0, の) + (f', Φ') can be forwarded, and taking the three pieces of understanding together, we arrive at the chain effect of churn.
The Chain Effect of Churn:
(U.I) Parameter updates cause the value and policy churn, (U.II) which further leads to the deviations in the action gradient and policy value; (U.III the churns and the deviations then bias following parameter updates.
As the cycle illustrated in Figure 2 the value and policy churn and the parame- ter update bias accumulate and can am- plify each other throughout the learning process. Intuitively, the parameter update chain could derail and fluctuate under the accumulating churns and biases, thus pre- venting stable and effective learning. We concretize our study on the consequences in the next section.
Ao, (Puaa) (Pru)
Parameter Update of Q and Policy
Cause
influence
Go (9,a))
D& (8,, Pena, ( Difiee, dees, [5f, (of,Fo.(s()) (s/333
Color Oeea--f ((,
Derive
Q-valuc Policy Churn
Devistion Action Gradient and Policy Value
Figure 2: Illustration of the logical cycle of the chain effect of the value and policy churn.
5 Reducing Value and Policy Churn in Deep RL
In this section, we show concrete learning issues caused by churn in typical DRL scenarios (Sec- tion S.T. followed by a simple plug-in method reduce churn and address the issues (Section (..2).
5.1 Consequences of the Chain Effect of Churn in Different DRL Scenarios
Since churn is involved in most DRL methods, as illustrated by Figure our study focuses on several typical DRL scenarios below.
5

---

Greedy action deviation in value-based methods Value-based methods like DQN train Q- network Oo and compute the policy by choosing the greedy action of Qo. A consequence of computing the action greedily is that changes in the values will directly cause changes in the action distribution [Schaul et al. [022]. Similarly to Eq. 4 this deviation can be formalized as: D2-(0,0 (0,0', BBt) = Bard Eee AA, max. Q=(3,a)) ( aarg max; Qe (3, a)). We suspect that this deviation introduces instability and hinders learning, and we focus on whether reducing churn can improve the performance of value-based methods.
Trust region violation in policy gradient methods Trust region plays critical role in many policy gradient methods for reliable and efficient policy updates. Proximal Policy Optimization (PPO) [Schulman et al. [017] uses a clipping mechanism as simple but effective surrogate of the trust
region for TRPO [Schulman et al. 2015]: Cl (Pola, Φ), 1 c, + c) and r(Quad, Φ)
xa(als)
(ala)
With respect to policy churn, even though the PPO policy conforms to the trust region for the states in
the current training batch, it could silently violate the trust region for other states, including previously
(a|s) ne (a|3) * 1
updated ones. Consider the policy update o + Ø', it is highly likely to have r(ø, Φ')
and thus r(oola, Φ') = Te (als)
Au [ā = r(Oola: (Ø) d') * r r(ool, 6). Since we have no information
about r(Ø; d'), there is no guarantee for the trust region e S r(owa, (²) S 1 + to be respected after churn. Intuitively, this implicit violation is hazardous and detrimental to learning.
Dual bias of policy value in Actor-Critic methods Deep AC methods interleave the training between the actor-network and the critic-network. Unlike the two scenarios above, where either the value churn or the policy churn raises a learning stability issue, we present the dual bias of policy value that stems from the bilateral effect of churn. The dual bias exists in the policy value as Qe (3, Ter (5)) * Qo(5, 6, o (5)). In the context of AC methods, the policy value is used for the target computation of the critic Tt + ?Qu (st+1, Tay (St+1)) and the optimization objective of the actor VoQu (s, Th (s)). Thus, the dual bias steers the training of the actor and the critic.
Given these negative consequences of chur,, a question is raised naturally: how can we control the level of churn to mitigate the issues without introducing complex trust regions or constraints?
5.2 A Regularization Method for Churn Reduction
In this section, we propose a regularization method to reduce value and policy churn, called Churn Approximated ReductloN (CHAIN). To combat the prevalence of churn's 's negative influence on DRI., our method should be simple to implement and easy to use with different RI. methods.
Based on the definitions of the value churn (Cq) and the policy churn (C₂,) in Section 4.2, we propose two corresponding loss functions Loc and LPc for churn reduction. Formally, for parameterized networks Q0,, To. at time t and reference batch B sampled from replay buffer, we have:
Lgc( Brat)
Σ B.e
(F, - (0,
(5)
Bret
8,0E
Loel Bre))=
Σ ze
de(TTo (3), To, (8))
(6)
where dz is policy distance metric, and we use mean square error KL divergence for deterministic or stochastic policies. Ideally, the regularization should be imposed on the post-update network parameters 0,41, Φt+1- Since they not available at time t, regularize fs, Φe and use 6,-1. Φ1-1 as the targets for a convenient and effective surrogate.
By minimizing Loc and LPc, we can reduce the value and policy churn and suppress the chain effect further. This allows us to use the churn reduction regularization terms along with standard RL. objectives and arrives at DRL with CHAIN finally:
minimize, L(Or, Buain) + AO Bret)
(8) where Birain Bie are two separate batches randomly sampled from D, and λo, X are coefficients that control the degree of regularization. CHAIN serves as a plug-in component that can be implemented with only a few lines of code modification in most DRL, methods. The pseudocode is omitted here, and we refer the readers to Algorithm I in the Appendix.
maximize, Hár,, Burain) - Toc Biet)

---

Automatic adjustment of X To alleviate the difficulty of manually selecting the regularization coefficients, we add a simple but effective method to adjust Aq. A adaptively during the learning process. The key principle behind this is to keep consistent relative scale (denoted by (3) between the churn reduction regularization terms and the original DRL, objectives. More precisely, by maintaining the running means of the absolute Q loss [ and the VCR term | ^o is computed dynamically
as XQ= = 3 |ماا which is similar for Ay. This is inspired by our empirical observations and the recent study on addressing the reward scale difference across different domains (Hafner et al. 2023)
Another thing worth noting is that CHAIN helps to mitigate the loss of plasticity via churn reduction. This connection can be established by referring to the NTK expressions in Eq. 2 reducing churn encourages kn, K to 0 and thus prevents the empirical NTK matrix from being low-rank, which is
shown to be consistent indicator of plasticity loss
[Lyle et al. 2024
6 Experiments
In the experiments, we aim to answer the following questions: (1) How large is the value and policy churn in practice, and can method effectively reduce chur?? (2) Does method's reduction of churn address learning issues and improve performance in terms of efficiency and episode return? (3) Does CHAIN also improve the scaling abilities of deep RL?
We organize our experiments into the four subsections below that correspond to the three DRL scenarios discussed in Section 5. as well as a DRL scaling setting. Our experiments include 20 online RL tasks from MinAtar, MuJoCo, DMC, and 8 offline RL datasets from D4RL, as well as popular algorithms,, i.c., DoubleDQN, PPO, TD3, SAC, IQL, AWAC. We provide the experimental details in Appendix C and more results in Appendix D
6.1 Results for CHAIN DoubleDQN in MinAtar
We use DoubleDQN (DDQN) [van Hasselt et al. 2016 as the value-based method and Mi- nAtar (Young and Tian 2019) as the experiment environments. MinAtar is an Atari-inspired testbed for convenient evaluation and reproduction. We build our DoubleDQN based on the official MinAtar code with no change to the network structure and hyperparameters. We implement CHAIN DDQN by adding few lines of code apply the value churn reduction regularization in the standard training of the Q-network (Eq, 77.. For CHAIN DDQN, Xo is set to 50 for Breakout and 100 for the other tasks. For CHAIN DDQN with automatic adjustment of Xo (denoted by the suffix 'Auto'), the target relative loss scale 3 is set to 0.05 for all the tasks.
First, to answer Question (1), we report the value churn and the greedy action deviation of DDQN in Figure 3. Each point means the average metric across randomly sampled states and the whole learn- ing process. As expected, we can observe that the value churn accumulates as more training updates take place, leading to the growth of greedy action deviation. With CHAIN, the churn and deviation are reduced significantly. We refer the readers to Figure for more statistics on the value churn.
COCS Chails Redlccdarns
Figure 3:
The value churn (left) and the greedy action deviation percentage (right) in Breakout w/ and w/o CHAIN.
Further, we show the learning curves of CHAIN DDQN regarding episode return in Figure 4 We can see that CHAIN consistently achieves clear improvements over DDQN in terms of both sample efficiency and final scores, especially for Asterix and Seaquest. Moreover, CHAIN (Auto) matches or surpasses the results achieved by manual coefficients, which supports Question (2) positively, In the next subsection, we evaluate how much CHAIN can improve policy gradient-based RL algorithms.
6.2 Results for CHAIN PPO in MuJoCo and DMC
Corresponding to the second DRL scenario discussed Section 5.1 we focus on the policy churn in Proximal Policy Optimization (PPO) [Schulman et al. 2017 and try to answer the first three questions for policy gradient-based RI. algorithms. We build the experiments on the public implementation of PPO from CleanRL (Huagg et al. 2022 and use the continuous control tasks in MuJoCo and
7

---

breakcutt
spate mnuddrr
Trewayy
DOON
CHAN DOON
CHANN DOUN (Muis)
(560)
Thocloge
Figure 4: The evaluation of CHAIN DoubleDQN in MinAtar regarding episode return. Curves and shades denote means and standard errors over six random seeds.
DeepMind Control (DMC) as the environments for evaluation. Following the same principle, we implement CHAIN PPO by adding the policy churn reduction regularization the standard PPO policy training (Eq. 6. with no other modification to the public PPO implementation.
First, to understand the level of chum, compare PPO and CHAIN PPO with different choices of X in terms of policy churn and episode return. In summary, we observed that PPO also exhibits clear policy churn, and CHAIN significantly reduces it throughout learning, which answers Question (1). Figure 10 shows the details for this analysis. Note that more policy churn makes it more likely to violate the trust region C. & r(o, d') discussed in Section 5.1 In turn, we also observed CHAIN PPO consistently outperforms PPO in Ant-v4 and HalfCheetah-v4 44 across different choices of Am
Further, we aim to answer Question (2) and evaluate whether CHAIN can improve the learning performance of PPO in terms of episode return. For CHAIN PPO (Auto), we set the target relative loss scale 3 to 0.1 for MuJoCo tasks and 0.02 for DMC tasks. The results for MuJoCo and DMC tasks are reported in Figure [ The results show that CHAIN PPO outperforms PPO in most cases with higher sample efficiency and final episode return, often significantly. We believe that our results reveal a promising direction to improve more trust-region- reaeeo- and constraint-bad based methods in DRL by addressing the issues caused by churn.
laalCheetsboot
laspenool
We Lae2
Blumnoic
1134
FPO
C-NNN PPD (Auns]
dm con.oootobbbuuoo Un c
dm, www.toticaaaaooeeeccccooo balano, sparso-00
di don.controilwwddiueeoo walk-bb
dm, wnw.cortowiiascuuup...oo ron-v0
Dileric Pootam
PPO
(ulr)
Sett
Rastt
an_contrdelio vo
di controbtiiggggooooo wallor
don_contrulddog run-v
dm, www_contrrildaggooo toot-e
dmn.conrrollogg Sctchw
Timaoo
Timiates
Timastep 2341
Tinatoe
Timaates
Figure 5: The evaluation of CHAIN PPO in MuJoCo and DeepMind Control (DMC) tasks regarding episode return. Curves and shades denote means and standard errors over twelve random seeds.
6.3 Results for Deep Actor-Critic Methods with CHAIN in MuJoCo and D4RL
Next, we continue our empirical study and evaluate our method for deep actor-critic (AC) methods. We separate our study into online and offline settings, presented below.
Online AC methods We use TD3 FFujimoto et al., 2018 and SAC (Haarnoja et al.) 2018 and MuJoCo environments based on the public implementation of TD3 and SAC from CleanRL. Since AC methods are bilaterally affected by churn, we consider two variants of CHAIN, either of which only applics the value churn reduction (VCR) the policy churn reduction (PCR).
8

---

For Question (I), we again find that both TD3 and SAC exhibit value and policy churn in all environments, and CHAIN- and CHAIN-PCR effectively reduce them respectively in Figure [T] and 12 For Question (2), Figure O shows the evaluation results regarding episode return. We can see that CHAIN-PCR often improves the learning performance, especially for Ant-v4; in contrast, CHAIN-VCR improves slightly. We hypothesize that this is because the policy interacts with the environment directly, and the target critic-network also helps to reduce the value churn due to its delayed synchronization with the online critic.
Due to the limitation of space, we refer the readers to Appendix D.3 for more results on churn reduction, the influence of different choices of λq A the results of combining VCR and PCR, as well as the effect of adjustmenntt of the regularization coefficient for TD3 and SAC.
llaspancl
1014
ClAAN TO3 INCZ
cd
Me
31C A5C D000
Figure 6: The evaluation of CHAIN TD3 and CHAIN SAC in MuJoCo regarding episode return.
Offline AC methods In Offline RL, a policy is trained over fixed dataset. We investigate if reducing churn can also improve the convergence of Offline RL. We use IQL Kostrikov et al.) [022] with D4RL Antmaze dataset [Fu et al. 2020 and AWAC (Nair et al., 2020) with Adroit for our demonstration due to their popularity and good performance in corresponding tasks, Concretely, we use the public implementation and benchmark scores for IQL. and AWAC from CORI To apply CHAIN to IQL and AWAC, we implement the regularization for value churn reduction (VCR, Eq. 7 and policy churn reduction (PCR, Eq. S separately by adding a couple of lines of code without any other modification. We use AS 1e3 for both CHAIN IQL (PCR) and CHAIN AWAC (PCR); λo 0.01 for CHAIN IQL (VCR) and 0.1 for CHAIN AWAC (VCR) across different tasks. The results are summarized in Table[annd 2
Table 1: Results for CHAIN IQL in Antmaze, with means and standard errors over twelve seeds.
Task
IQL
CHAIN IQL (PCR)
CHAIN IQL (VCR)
AM-umazc-v2
77.00 5.52
84.44 + 3.19
83.33 = 2.72
AM-umaze-diveree-y
54.25 5.54
62,50 3.75
71.67 = 7.23
AM- (M-mediuum-pla--22
65.75 + 11.71
72.50 + 2.92
70.00 3.33
AM- -iverss---
73.75 = 5.45
76.67 1: 4.51
66.67 = 3.79
AM-large-play--2
42.00 = 4.53
50,00 + 4.56
43.33 = 4.14
AM-large-diverse-
30.25 - 3.63
26.67 3.96
31.67 = 2.31
Table 2: Results for CHAIN AWAC in Adroit, with means and standard errors over twelve seeds.
Task
AWAC
CHAIN AWAC (PCR)
CHAIN AWAC (VCR)
pen-human-vl
81.12 + 13.47
99.72 + 2.04
97.37 3.51
pen-cloned-vl
89.56 + 15.57
95.49 + 2.34
96.66 2.54
We observe that both CHAIN PCR and CHAIN VCR improve the scores for IQL and AWAC in most Antmaze and Adroit tasks. We hypothesize that CHAIN suppresses churn in the training of value and policy networks, thus reducing the bias caused by churn in parameter updates. One thing here that differs from TD3 and SAC considered in the online setting that the policy network of IQL has no impact on the training of the value networks since the value networks (i.c.. Q and V) are trained purely based on in-sample data without accessing a' TT)(s') Thus, although IQL does not exhibit a chain effect explicitly, the policy and value networks of IQL still have churns, which are reduced by CHAIN in this case. We provide a further empirical study in Appendix DD4
9

---

6.4 Scaling DRL Agents with CHAIN
is widely known that scaling DRL agents up is challenging. Naively scaling DRL agents by widening deepening the conventional MLP networks straightforwardly often fails and could even lead to collapse. Here, we investigate the relationship between churn and scale for DRL agents, as well as the effeet of CHAIN on boosting scaling performance, to answer Question (3), We take PPO and MuJoCo tasks as the exemplary setting and scale up both the policy and value networks by scale-up ratio within (2,4, 8, 16) via widening or deepening- Note that the default network architecture (i.e., when the scale-up ratio equals one) for both the policy and value networks is a two-layer MLP with 256 neurons for each layer, followed by an output layer.
As expected, we observed that the performance of PPO degraded severely as the scale-up ratio increased, as shown by the solid gray lines in Figure 7 Inspired by the prior study [Obando-Ceron et al. 2024], we found using decreased learning rate as 1r sqrt (scale-up ratio) alleviates the degradation of PPO scaling to some degree (shown by the solid red lines). We then use the learning rate setting below by default. From the perspective of churn, wc also observed that scaling PPO escalates the scale of the policy churn in PPO. More results can be found in Appendix [D.5 Therefore, we then evaluate the effect of CHAIN in this scaling setting. The results are shown in Figure 7 with dashed lines. By comparing the lines in the same color, we found that CHAIN improves the learning performance of PPO across almost all scale-up ratios and the two learning rate settings.
Ant.v4 (2M)
Humancid v4 (2M)
Walker2d (2M)
1000-
Porfrmsncce saree
25.00
Performace Wocnet
2002
Peffrmuncc S-oret
4030
2020
1502
-920
2003
3020
1000
PPO
520-
CHAIN
PPO scrt.
2002
2020
500
Scale Up Radio
Scale Up Rabio
Scale Up Ratio
Figure 7: The results regarding episode return for scaling PPO via widening. CHAIN helps to scale almost across all the configurations. Similar results can be found for widening scaling in Figure 20
In addition, we extend the training horizon from 2M to 10M for Ant, Humanoid, and Walker2d. The results are reported in Table 3 For both widening or deepening cases, CHAIN helps to scale up PPO and achieves clear improvement in terms of episode return. Comparatively, scaling by widening slightly outperforms deepening, which echoes the observation in [Ota et al. 2020) to some extent.
Table 3: Scaling PPO with CHAIN. Means and standard errors of final episode return over six seeds.
Alg. (scale)
Ant (10M)
Human. (10M)
Walker2d (10M)
PPO
2238.45 + 256.07
1620.45 + 212.10
3316.77 = 269.42
PPO (4x wider)
3013,95 + 223.77
2998.18 + 237.95
3795.05 # 208.17
CHAIN PPO (4x wider)
4916.66 + 109.01
4830.58 + 231 42
4668.16 + 234.45
PPO (4x deeper) CHAIN PPO (4x deeper)
2777.62
136.78
3489.47 + 166.28
2845.6 260.02
3760.46 158.29
4090.80 + 187.01
3242.83 173.54
PPO (8x wider) CHAIN PPO (8x wider)
4235.63
209.68
3198.01 + 344.71
4335. 123.12
5592.94
205.23
5246.15 + 130.73
5161.42 343.40
PPO (8x deeper)
3364.46 + 173.25
2780.67 + 304.10
3057,88 89 + 299.61
CHAIN PPO (8x deeper)
4278.76 + 122.61
4434.48 + 144.92
3324.29 + 255.88
Our results indicate that uncontrolled churn could be a possible reason for the scaling issue of DRL agents, and CHAIN improves scaling by reducing chur effectively. Though appealing, CHAIN does not fully address the scaling issue per se, and achieves sub-linear scaling on DRL agents.
7 Conclusion
In this paper, we conduct a formal study of churn in a general view and present the chain effect of value and policy churn. The chain effect indicates a compounding cycle, which biases parameter updates throughout learning. We propose an easy-to-implement method for value and policy churn reduction. Our experimental results demonstrate the effectiveness of our method in reducing churn and improving learning performance over range of DRL environments and algorithms.
10

---

Acknowledgements
We want acknowledge funding support from NSERC, FQRNT and CIFAR and compute support from Digital Research Alliance of Canada, Mila IDT, and NVidia.
References
Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos C. Machado. Loss of plasticity in continual deep reinforcement learning. arXiv preprint, arXiv:2303.0750) 2023.
J. Achiam, E. Knight, and P. Abbeel. Towards characterizing divergence in deep q-learning. arXiv preprint, arXiv:193.00889, 2019.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Deep reinforcement learning at the edge of the statistical precipice. NeurIPS, 2021.
Marc G. Bellemare, Will Dabney, and Rémi Munos. distributional perspective reinforcement learning. ICML, 2017.
Emmanuel Bengio, Joelle Pineau, and Doina Precup. Interference and generalization in temporal difference learning. ICML, 2020.
G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv preprint, 660iv:666..015444 2016.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. Randomized ensembled double q-learning: Learning fast without model. In ICLR, 2021.
Zaiwei Chen, John-Paul Clarke, and Siva Theja Maguluri. Target network and truncation overcome the deadly triad in q-learning. arXiv preprint, arXiv:2203.02622 2022.
Karl Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient. In ICML, 2021.
W. Dabney, A. Barreto, M. Rowland, R. Dadashi, Quan, M. G. Bellemare, and D. Silver. The value-improvement path: Towards better representations for reinforcement learning. In AAAI. 2021.
Shibhansh Dohare, J. Fernando Setnandez-Garcin Parash Rahman, Richard S. Sutton, and A. Rupam Mahmood. Maintaining plasticity in deep continual learning. arXiv preprint, arXiv:2306. 13812, 2023.
Pierluca D' D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre- Bacon, Mare G. Bellemare, and Aaron C. Courville. Sample-efficien reinforcement learning by breaking the replay ratio barrier. In ICLR, 2023.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: datasets for deep data-driven reinforcement learning. arXiv preprint, arXiv:2004.07219 2020.
S. Fujimoto, H. v. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In ICML, 2018.
Sina Ghiassian, Banafsheh Rafiee, Yat Long Lo, and Adam White. Improving performance in reinforcement learning by breaking generalization in neural networks. In AAMAS, 2020.
Haarnoja, Zhou, Abbeel, and Levine. Soft actor-critie: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In ICML, 2018.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy P. Lillicrap. Mastering diverse domains through world models. arXiv preprint, arXiv:2301.04 2023.
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Gheshlaghi Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In AAAI, 2018.

---

Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Ki- nal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1-18, 2022.
Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Boehmer, and Shimon Whiteson. Transient non-stationarity and generalisation in deep reinforcement learning. In ICLR, 2021.
Steven Kapturowski, Victor Campos, Ray Jiang, Nemanja Rakicevic, Hado van Hasselt, Charles Blundell, and Adrià Puigdomènech Badia. Human-level atari 200x faster. In ICLR, 2023.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In ICLR, 2022.
A. Kumar, R. Agarwal, D. Ghosh, and Levine. Implicit uudem-paramettriizatoonn inhibits data- efficient deep reinforcement learning. In ICLR, 2021.
A. Kumar, R. Agarwal, T. Ma, A. Courville, G. Tucker, and S. Levine DR3: value- deep reinforcement learning requires explicit regularization. ICLR, 2022.
Aviral Kumar, Abhishek Gupta, and Sergey Levine. Discor: Corrective feedback in reinforcement learning via distribution correction. In NeurIPS, 2020.
Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, and Dmitry P. Vetrov. Controlling over- estimation bias with truncated mixture of continuous distributional quantile critics. In ICML, 2020.
Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White. Maxmin q-learning: Controlling the estimation bias of q-learning. In ICLR, 2020.
Qingfeng Lan, Yangchen Pan, Jun Luo, and A. Rupam Mahmood. Memory-efficient reinforcement learning with value-based knowledge consolidation. Transaction on Machine Learning Research, 2023.
Jongmin Lee, Wonseok Jeon, Byung-Jun Lee, Joelle Pineau, and Kee-Eung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In ICML, 2021.
Boyan Li, Hongyao Tang, Yan Zheng. Jianye Hao, Pengyi Li, Zhen Wang, Zhaopeng Meng, and Li Wang. Hyar: Addressing discrete-comtinuuus action reinforcement learning via hybrid action representation. In ICLR, 2022.
P. Lillicrap. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, Continuous control with deep reinforcement learning. In ICLR, 2015,
Vincent Liu, Adam White, Hengshuai Yao, and Martha White. Towards practical measure of interference for reinforcement learning. arXiv preprint, arXiv:2007.038077 2020.
Vincent Liu, Han Wang, Ruo Yu Tao, Khurram Javed, Adam White, and Martha White. Measuring and mitigating interference in reinforcement learning. In ICML. 2023.
Tyler Lu, Dale Schuurmans, and Craig Boutilier. Non-delusional q-learning and value-iteration. In NeurlPS, 2018
C. Lyle, M. Rowland, and W. Dabney. Understanding and preventing capacity loss in reinforcement learning. In ICLR, 2022a.
Clare Lyle, Mark Rowland, Will Dabney, Marta Kwiatkowska, and Yarin Gal. Learning dynamics and generalization in deep reinforcement learning, In ICML, 2022b.
Clare Lyle, Zeyu Zheng, Khimya Khetarpal, Hado Hasselt, Razvan Pascanu, James Martens, and Will Dabney. Disentangling the causes plasticity loss in neural networks. arXiv preprint, arXiv:2402. 2024.
Y. Ma, H. Tang, D. Li, and Z.. Meng. Reining generalization in offline reinforcement learning via representation distinction. In NeurIPS, 2023.
12

---

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-333 2015.
Ofir Nachum, Bo Dai, llya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schaurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint, arXiv: 191..02044, 2019.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint, K0X5:20006.09359 2020.
Michal Nauman, Michal Bortkiewicz, Mateusz Ostaszewski, Piotr Milos, Tomasz Trzcinski, and Marek Cygan. Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning. arXiv preprint, MXXxx224030005444 14, 2024.
E. Nikishin, M. Schwarzer, P. D'Oro, P. Bacon, and A. C. Courville. The primacy bias in deep reinforcement learning. In ICML,, Proceedings of Machine Learning Research, 2022.
Johan S. Obando-Ceron, Aaron C. Courville, and Pablo Samuel Castro. In deep reinforcement learning, pruned network is good network. arXiv preprint, arXiv:2402.12479, 2024.
Georg Ostrovski, Pablo Samuel Castro, and Will Dabney. The difficulty of passive learning in deep reinforcement learning. In NeurlPS, 2021.
Kei Ota, Tomoaki Oiki, Devesh K. Jha, Toshisada Mariyama, and Daniel Nikovski. Can increasing input dimensionality improve deep reinforcement learning? In ICML, 2020.
Yangchen Pan, Kirby Banman, and Martha White. Fuzzy tiling activations: simple approach to learning sparse representations online. In ICLR, 2021.
Alexandre Piché, Valentin Thomas, Joseph Marino, Rafael Pardinas, Gian Maria Marconi, Christopher Pal, and Mohammad Emtiyaz Khan. Bridging the gap between target networks and functional regularization. Transactions on Machine Learning Research, 2022.
Roberta Raileanu and Rob Fergus. Decoupling value and policy for generalization in reinforcement learning. ICML, 2021.
T. Schaul, A. Barreto, J. Quan, and G. Ostrovski. The phenomenon policy churn. arXiv preprint, arXiv:2206.00730, 007300 2022.
Tom Schaul, John Quan, loannis Antonoglou, and David Silver. Prioritized experience replay. In ICLR, 2016.
John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In ICML, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint, 1tXiv:1707.06347. 2017.
D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. A. Riedmiller. Deterministic policy gradient algorithms. In ICML, 2014.
G. Sokar, Agarwal,
Castro, and U. Evei. The dormant neuron phenomenon deep reinforee- ment learning. arXiv preprint, 11XX:22302.12902 2023.
Dijia Su, Jayden Ooi, Tyler Lu, Dale Schuurmans, and Craig Boutilier. Conqur: Mitigating delusional bias in deep learming.. In ICML, 2020.
R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. IEEE Transactions on Neural Networks, 16:285-220 28665-26,111 1988.
H. Tang, Z. Meng, J. Hao, Chen, D. Graves, D. Li, C. Yu, H. Mao, W. Liu, Y. Yang, W. Tao, and L. Wang. What about inputting policy value function: Policy representation and policy-extended value function approximator. In AAAL 2022.
13

---

Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. de Las Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefraneq. T. Lillicrap, and M. A. Riedmiller. Deepmind control suite. arXiv preprint. arXiv: 1etv:1801.000 1008000600) 2018.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double learning. AAAI, 2016.
Hado van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad. arXiv preprint, arXiv: 1.Xiv:1812.02643. 2018.
Guowei Xu, Ruijie Zheng, Yongyuan Liang, Xiyao Wang, Zhecheng Yuan, Tianying Yu Luo, Xiaoyu Liu, Jiaxin Yuan, Pu Hua, Shuzhen Li, Yanjie Ze, Hal Daumé III, Furong Huang, and Huazhe Xu. Drm: Mastering visual reinforcement learning through dormant ratio minimization. In ICLR, 2024.
Ge Yang, Anurag Ajay, and Pulkit Agrawal. Overcoming the spectral bias of neural value approxima- tion. In ICLR, 2022.
Kenny Young and Tian Tian. Minatar: An atari-inspired testbed for more efficient reinforcement learning experiments, arXiv preprint, MrXIiv:1933.05 2019.
Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. In ICLR, 2021a.
Ruiyi Zhang, Bo Dai, Lihong Li, and Dale Schuurmans. Gendice: Generalized offline estimation of stationary values. ICML, 2020.
Shangtong Zhang, Hengshuai Yao, and Shimon Whiteson. Breaking the deadly triad with target network. In ICML, 2021b.
14

---

A Limitations
Our work is limited in several directions below and we expect further studies on these points in the future.
First, the theoretical analysis of the chain effect under concrete assumptions remains to be explored. Other perspectives that may influence churn, such as network structure, representation learning. and experience replay are not considered this work, which are worthwhile to study in the future.
Moreover, although we provided a simple method to adjust the regularization coefficients dynamically throughout learning by keeping consistent relative loss scale, is not suffi- cient for us to use the same hyperparameter for different domains. We believe that using normalization techniques to unify the scales in different domains necessary address this point, similar to the work in [aater et al. 2023).
Besides, for algorithms that involve the learning of both policy and value (e.g., deep AC methods), the implementation of CHAIN in different problems faces the question of choosing the best option among PCR, VCR and DCR. For this, we expect to develop a better method integrate the effect of PCR and VCR in the future.
Another remaining problem is the lack of an in-depth understanding of the positive and negative effects of churn on the generalization of DRL agents, which could drive new methods that better leverage the potential of churn.
B Additional Formal Analysis
B.1 The NTK Expressions of Two Types of Deviation
The NTK expression for the action gradient deviation DJ, is straightforward to obtain by plugging in the NTK expression for the Q-value churn:
D._ (0,0') V VaQo (5, (), VaQo (0.ass))aa=xt a)la=x(3)
V. Qo Qa (a),x(8))=x(1))
Co(0,0)
(s,
02 (Q0(5..)
A Apla=n(3)
200a
And for the policy value deviation Dz the NTK expression is obtained by performing Taylor expansion of (5) and plugging in the NTK expression for the policy churn:
D5 (VaQo (a)=xo()))))
(T) - ma()) O(1 (T) To 112)
(o.04))
C=(o.0)
(VaQo (3, a) (e)) his 8) V la=
V (3) Vo (B, A T
DPG Fe at
The NTK expression of the two types of deviation above indicates that action gradient deviation and policy value deviation are mainly influenced by the second-order partial derivatives of Qo and the deterministic policy gradient (DPG) of T. at the state ş considered, in addition to the parameter update. An interesting observation here is, policy value deviation of policy churn has an implicis optimization effect (while altered by A&) for the reference states (i.e.. 5) although the parameter update Φ + ø' is performed for the training states (i.e., s). We observe that D3 is positive overall in our empirical investigation later (Appendix (D). However, careful considerations are needed in the future because the implicit optimization could be delusional as churn also oceurs in Qo through the learning process.
15

---

Aguu
Aa
Co 02)
Qe
Q0en2
n Bcal
Age-t
Figure 8: An illustration of the parameter update chain with the value and policy churn. Blue and green arrows denote iterative updates of the value network and policy network. The value and policy churn and their deviations are marked in red.
B.2 Complete Discussions and Derivations for Section 4.3
Recall conventional gradients Ag. As value network and policy network:
as)s,a) (o)
4x()) V. (a)=((()) (#)
where 60(s, = (s))))
Now let us re-consider the gradients by taking into consideration the value and policy churm, denoted by Âe, Ag. Our purpose is formulate the difference between Âg, As and De. Dg as functions of the policy and value churns, as well as their derivatives.
More derivatives of Cq. Cr Before looking into the gradients Åo, Åg. need three more defini- tions for network parameter gradient deviation caused by value and policy churn, and action gradient deviation caused by policy churn, during parameter update + 0', db H'.
Q-network Gradient Deviation of Value Churn:
= V (3,
VoQa(3,0) a).
Policy Network Gradient Deviation of Policy Churn: DE (0, (3)) Vo (3)
Action Gradient Deviation of Policy Churn: DE, (2. V ((s 12
VaQ(3, ())=
Note DE, (d, d', {s}) denotes the action gradient caused by the policy churn for the Q function affected by the churns Qe rather than Qo. DJ, is a further consequence of policy value deviation
caused by the policy churn "7
To shed light on the long term effect of churn, we depict typical iterative update scenario in Figure 8 where the Q-network and policy network update with corresponding gradients chain. Different from conventional analysis, we explicitly consider the value and policy churn and study how they affect the chain of parameter updates.
As in Section 4.3 we focus on the segment (0- - (0, (0', (0', Φ') on the chain of update. The churns occurred during the past update (0 o )-) (), which should further affect the update about to perform at present (0, σ) + (0', o¹). Concretely, the churns and the deviations affect the following aspects:
(1) Q-value estimate and (2) action selection in both TD error and policy objective, and (3) the gradient of network parameter. Now we are ready to deduce the parameter update under the effect of the value and policy churn below. Note that we use under the terms that are affected by
16

---

value and policy chur::
L0(s, a) - Volg(o - -- (o(s,)))
+ volo(s,a)) (0000600000000000000+++000000100 -
Qe(s',a))
(Q((s' - (Q(6)))))(((0)))((((000)))))))))
(Volo(s,a) D (0 (.,))) (G00:
-C,(07,0,18.a)))
VVQo(s..)do(s..c)
(x()) = ( ( (zT
(VoTo(s) (x) - (V.lo)))) (.0l))))) -V.(((()))-VQQ((s..)))
(VaQo(s,a) v=Qo(s,a)) (s)
(0 (TT((s)+ + ( の) VaQo(s,a De (0- (==m((+)
Vo=o(s)V a= (()))) (T) (0 + (0 0, (s,a))) la=T((s)
(6 ( o) (v.Qo(s, a) DJ. ( (6)) D ,0 (8, la=xa(+)
As result, we can find that the value and policy churn, as well as the deviations derived, introduce biases in the parameter updates.
Recall that the parameter updates cause the churns constantly, the analysis on the update segment (0 o + (0, Φ) (0', d') can be forwarded and leads to the cycle illustrated in Figure 2 (1) parameter update causes the value and policy churn Cq. Cr. which (2) further deviates the action gradient and policy value D7, D&, (and the other deviations); (3) the churns and the deviations then bias following parameter updates with Âo, Âg. Consequently, the value and policy churn and the parameter update bias accumulate and can amplify each other throughout the leamnnn process.
Apparently, the long-term chain effect is intricate the training process stochastic and the churns are influenced by various factors, e.g., network structure, learning objective, and data distribution. We leave theoretical studies under concrete assumptions in the future.
C Experimental Details
C.1 Compute Resources and Time Cost
We use Nvidia V100 GPU for our experiments. The additional computation introduced by CHAIN lies in the sampling and training with a second batch of data for the churn reduction regularization. In essence, it just increases the batch size by 2 to use CHAIN, resulting in constant change in complexity with respect to the default DRL algorithm. This is a small price to pay to reduce the churn and increase policy performance. In practice, we also observe similar wall-clock time cost for standard DRL methods and their CHAIN versions.
More importantly, since CHAIN often brings higher sample efficiency, i.e., achieving the same level of score with fewer interaction steps, this implies that CHAIN accelerates learning and achieves good performance earlier. Concrete examples can be found in experimental results.
17

---

C.2 Empirical Metrics for the Investigation of the Chain Effect
To investigate the extent of the value churn and the policy chur, we compare the output changes between current networks (fc. ox) and past network versions (0 (0- E ( 0.-1))==1
For policy-based methods, we study TD3 and SAC for MuloCo. We compute the value churn (Co... (QQ...). the policy churn (C,,) and the value deviation of policy churn (Dz) throughout learning.
For value-based methods, we compute the percentage of the greedy action deviation (D., ) the value churn of greedy action (Cq..) and the value churn of all actions ((q,, These metrics are summarized in Table 4
Table 4: Churn and deviation metrics used in our experiments. 8t, di and f_, Φ are current networks and previous networks. The metrics are averaged over 8, a reference buffer.
The Metrics for Policy- based Methods
Co... .0)
Qo, a) Qa- (5, (for TD3, SAC) (5, Qa (5. a)l (for TD3, SAC)
aq. (0 9b)
I (s) - Tg- (+))]] (for TD3, PPC
Cx
KL(Ta, (115), F (:18) (for SAC)
(0 (b))
Qo, (3, Fe, (8)) Qo, T (F))
The Metrics for Value-hased Methods
(1 .0)
A Enea 1Q0, ()) a) IaaXX (3, a') max, Qa (5, a) D (0)) [(arg Qa- (8.0)) (arg max, Qo, (s,a))
Q0- (8, a)]
Co (0)
We use N 50 and N 20 for MuloCo and MinAtar environments and compute the metrics at an interval of 1k parameter updates. For each type of metric in Table 4 and update number i {1, ..... N). we compute the mean of the quantities throughout learning. We refer the readers to Figure and 12 for the results in Appendix D
C.3 Code Implementation
We the public implementations of PPO, TD3 and SAC in CleanRi our codebase. The actor and critic networks are two-layer MLPs with 256 units for each layer. For DoubleDQN, we modified the DQN implementation provided in the official code of MinAtar paper with no change to the network structure, recommended hyperparameter choices, etc. Hyperparameters are listed in Table 5 Table 6 and Table For the experiments the offline RL setting, we use the public implementation and benchmark scores for IQL. and AWAC from CORI
One thing to note is that the data in the training batch and the regularization batch should be non- overlapping in principle, but we found simply sampling two random batches from the replay buffer independently works well. From the probabilistic perspective, the overlap could happen at a low probability, which is determined by the batch size and the size of the replay buffer.
We make change to the statc/observation and reward of MuJoCo and MinAtar environments. No additional tricks like state normalization, reward normalization are used in our experiments.
C.4 Other Discussions
Discussion the Data Batches Used in Training and Anlysis We use separate sets of data for churn reduction regularization (i.e., the regularization set) and churn investigation/evallttie (i.e., the actual reference set), and they randomly sampled each network update. In other words, if count in the regular batch for training, we have three separate batches in total: (1) a regular training
http::///jhhib..cmm//eeyyyy///yp httpg://ggiaabb.cc//titooot..t.
18

---

batch for standard RL loss computation, (2) regularization batch for churn reduction loss, and (3) reference batch for chum evaluation (optional) throughout the learning process. Note that the reference batch is only used for churn evaluation and does not influence learning.
Table 5: Hyperparameters of DoubleDQN used in MinAtar environments. The values of conventional hyperparameters taken from the recommended values in (Young and Tian 2019.
DoubleDQN Hyperparameters
Learning Rate
3e
Training Interval
step
Discount Factor (?)
0.99
Hard Replacement Interval
1000 steps
Replay Buffer Size
0.5M
Batch Size
32
Initial
1.0
End
0.1
Decay Steps
0.5M
Initial Random Steps
10k
Value Churn Reduction Hyperparameter
50 for Breakout
Value Regularization Coefficient (Aq)
100 for others
Target Relative Loss for Auto Xo
0.05
Table 6: Hyperparameters of PPO used in MuJoCo environments. hyperparameters are taken from the recommended values in CleanRL.
The values of conventional
PPO Hyperparameters
Learning Rate
3e
Training Interval
2048 steps
Discount Factor (7)
0.99
GAE Parameter (A)
0.95
Num. of Minibatches
32
Update Epoch
0.1 for MuJoCo tasks
Clipping Range Parameter (c)
0.2 for DMC tasks
Policy Churn Reduction Hyperparameter
5000 for Ant-v4
Policy Regularization Coefficient (A₂)
50 for other MuJoCo tasks
0.1 for MuJoCo tasks
Target Relative Loss 3 for Auto >
0.02 for DMC tasks
19

---

Table 7: Hyperparameters TDD and SAC used in MuJoCo environments. The values of conventional hyperparameters are taken from the recommended values in CleanRL. means 'not applicable'.
TD3 & SAC Hyperparamelers
Hyperparameters
TD3
SAC
Actor Learning Rate
3e
Critic Learning Rate
3e
3e
Actor Training Interval
2 steps
step
Critic Training Interval
step
step
Exploration Noise
N 0.1)
Target Action Noise
(.0.0.2) 0.2)
Target Action Noise Clip
0.5
Discount Factor (n)
0.99
0.99
Soft Replacement Ratio
0.005
0.005
Initial Random Steps
5k
5k
Replay Buffer Size
IM
IM
Batch Size
256
256
Optimizer
Adam
Adam
Churn Reduction Hyperparameters (refer to the study in Fig.
Policy Regularization Coefficient (Ax) Value Regularization Coefficient (Λy)
(0.1, 1, 20
{le_² 5e
(0.1, 0.5, 1.0)
Target Relative Loss 3 for Auto X,
5e
Algorithm Deep RL with Churn Approximated ReductloN (CHAIN).
1: Initialize the policy network Ts and the value network Qe with parameters o, 0 if they exist
2: Initialize an empty buffer D
3: set churn reduction hyperparamters λq, A-
4: (Optional) Set target relative loss scale for auto-adjustment of λq A=
5: for iteration t = 1, 3, do
6: // Interact with the environment and collect samples
7: Rollout the policy with т& Qu and store interaction data in D
8: // Perform value and policy network learning
9:
if time to update then
10:
Sample training batch Buain and a reference batch B from Update Qo with L(9, Busin), Loc(9, Byet) by Eq 7
11:
12:
Update To with L(ó, Brai). ). Lpc (ò, B,e)) by Eq.
13:
(Optional) Re-calculate Xq, X according 3 (reter to Section 5.2
14:
end if
15:
end for
20

---

D Complete Results
D.1 More Empirical Analysis the Value Churn in DoubleDON
In Figure shows the statistics of the three metrics defined in Table 4 Horizontal axes show the number of parameter updates after which the statistics are computed. Rach point is obtained by averaging all the quantities throughout learning. Curves and shades denote means and standard errors across six random seeds.
We can observe that the amount of value churn (Gq,) accumulates as the update number increases. Although there does not exist an explicit policy network, the percentage of greedy action deviation (CQ..)) ) goes up over 20%. However, different from the case for TD3 and SAC, the value of greedy action (D..) ) decreases and the value churn (Co,) exhibits a similar trend.
Updar
Updrn
(a) Percentage of Greedy Action Change
(b) Value Change Greedy Action
(c) Value Change of All Actions
Figure 9: Different statistics on value churn of DoubleDQN in MinAtar. Horizontal axes are numbers of parameter updates after which the statistics are computed. Each point is obtained by averaging all the quantities throughout learning. Curves and shades denote means and standard errors across six random seeds.
D.2 More Results of CHAIN PPO
Figure 10 provides the learning performance of CHAIN PPO four MuJoCo tasks. Moreover, also report the conventional PPO policy loss and the regularization loss Loc(O) during the learning process. Figure also shows the results for different choices of the hyperparameter X
From the results, we can observe: (1) PPO exhibits clear policy churn (the second column). One thing to note is that the fading of the policy churn is due to the decay of the learning rate in CleanRL's 's PPO implementation. CHAIN (2) reduces the policy churn PPO effectively, and (3) shows performance improvement in Ant and HalfCheetah and comparable performance in Hopper and Walker2d. For the hyperparameter choice, in practice, we found the churn reduction coefficient between 1e3 and le4 works well for Ant, while 50 works best in HalfCheetah.
We hypothesize that the choice of X is likely to related to the scale difference between policy loss (Ø) (the third column) and regularization term Lpc(O) (the fourth column). This motivates the proposal of our method for automatic adjustment of the regularization coefficient presented in Section 5.2
21

---

-0.1
(a) Ant
1944
HalfChcetah
Pedel mo
Timeniidall
Thr
(c) Walker2d
(d) Hopper
Figure 10: Results for CHAIN PPO, including the learning performance, the amount of the policy churn, the loss of conventional policy training, and the loss of regularization.
D.3 More Results of CHAIN for Deep AC Methods
We refer the readers to the following figures and tables for concrete additional results:
Figure [1 and Figure 12 show how the value churn reduction (VCR) and policy churn reduction (PCR) take effect during the learning process of TD3 and SAC. The four metrics defined in Table 4 are used.
Figure 13 and Figure 14 show the results of different choices of λo and X when VCR, PCR, or VCR+PCR (i.e., denoted by DCR) in CHAIN TD3 and CHAIN SAC.
Figure 15
shows an overall comparison among VCR, PCR, DCR when using either of HAIN TD3 and CHAIN SAC. We also provide the comparison with Reliable
them in
Metrics Agarwal et al., 2021 in Figure 17 and Figure 18
Figure 16 shows the evaluation of the automatic adjustment method of > for CHAIN TD3 in four MuJoCo tasks.
The Efffet of CHAIN-VCR and CHAIN-PCR in Reducing Churn For TD3 and SAC, the amount of value churn (QIq...) and policy churn (C,) increases the update number increases. This is expected as the chain effect indicates the accumulation of churn and deviation. Besides, the churn saturates after sufficient number of updates the networks. We hypothesize the target network helps cap the amount of churn.
Moreover, we observe positive policy value deviation action churn (D)). which matches our discussion implicit optimization effect of the policy chur in Appendix in contrast, the value chur (Cq... fluctuates above and below 0.
22

---

According to the relative scale, TD3 and SAC show less value and policy churn than DoubleDQN, This is due the structure independence of actor and critic networks. Another possible reason is that MuloCo locomotion may have smoother underlying problem dynamics than the MinAtar.
OMTO Dest
110m V
Updia
(a) HalfChectah-vd
(1 111
(b) Hopper-v4
art
200
urich (u
(c)
Walker2d-v4
15300
(d) Ant-v4
Figure 11:
Churn reduction of TD3 in MuJoCo. The 2nd-4th columns report the value of the four metrics defined in Table Horizontal axes are numbers of parameter updates after which the statistics are computed. Each point is obtained by averaging all the quantities throughout learning.
Discussion the Effect of CHAIN-VCR, -PCR, -DCR in Improving Episode Return In addition to the two variants of CHAIN introduced the main body of this paper, we introduce the third one, i.e., Double Churn Reduction (DCR), which corresponds to applying VCR and PCR the same time. The related results are reported in Figure 13
15
For CHAIN-VCR and CHAIN-PCR, formance, especially for Ant-v4. In contrast, CHAIN- VCR improves slightly. We hypothesize that this is because policy interacts with the environment directly and the target critic-network also helps to cap the value churn. Between TD3 and SAC, CHAIN-PCR works better for TD3 rather than SAC. We hypothesize that the variation (regarding the scale and range) is higher in optimizing the Maximum-Entropy objective and KL--based PCR term together for SAC than in optimizing the Q objective and 1.2-based PCR term for TD3. Another hypothesis is that the Maximum- Maximum-Entropy nature of SAC prefers the encouragement of more stochasticity in policy.
found that CHAIN-PCR often improves the learning per-
For CHAIN-DCR, we found that it is not easy to gain an immediate additive improvement regarding episode return when using the same hyperparameter choices Xo, A from both sides. We suggest that this reflects the intricate nature of the chain effect. As mentioned in Section 7 this points out the limitation of this work in aspects like the in-depth theoretical analysis the long-term chain effect, the lack of automatic coefficient adjustment, and the finer merrneraieed understanding of the positive and negative effects of churn. We leave these directions for future work.
23

---

18 m100M
Iivestip (9e0
HalfCheetah-v4
(b) Hopper-v4
(c) Walker2d-y4
1080
(d) Ant-v4
Figure 12: Churn reduction of SAC in MuJoCo. The 2nd-4th columns report the value of the four metrics defined in Table Horizontal axes are numbers of parameter update after which the statistics computed. Each point is obtained by averaging all the quantities throughout learning.
4000
120
100
Tirampisset
Tineroo SMl
Tewis dal
Themig Cal
(a) Value Churn Reduction (VCR)
2000
153 20 Sued
R010
(b) Policy Churn Reduction (PCR)
Wh Sue (u4e
11 21
Trancag (all
(c) Double Churn Reduction (DCR = VCR + PCR)
Figure 13: Hyperparameter choices for the value and policy churn reduction regularization for TD3 in MuloCo. Curves and shades denote means and standard errors across six random seeds.
24

---

(a) Value Churn Reduction (VCR)
RC1
6ic% RCV0 de
(b) Policy Churn Reduction (PCR)
(c) Double Churn Reduction (DCR = VCR + PCR)
Figure 14: Hyperparameter choices for the value and policy churn reduction regularization for SAC
in MuJoCo. Curves and shades denote means and standard errors across six random seeds.
1C 01 30-0
Alceroo 20-3)
Figure 15: An overall comparison among using value churn reduction, using policy churn reduction, and using them both for TD3 and SAC.
Kileera will
Wa Wt
Arl
1000
CMA TDO 6.33
CHA TDO 0 6.32
CHA 0g 3.01
CHAN TDO 12, -21
244 253 302
(66 25
Figure 16: The results for auto-adjustment of X for CHAIN TD3 in four MuJoCo tasks, with means and standard errors across six seeds. The target relative loss scale 3 is set to 5e-5, CHAIN (Auto) achieves comparable performance with manually selected coefficients and improves TD3.
25

---

The Effect of Automatic Adjustment of Xx Figure 16 shows the evaluation of the automatic adjustment method of ' for CHAIN TD3 in four MuJoCo tasks. We can observe that CHAIN (Auto) achieves comparable performance with manually selected coefficients and improves TD3, similarly the conclusions we found in Figure 5
T23 ocn
10a ICR
703 WOR
0.0
6.2
0-2
0.2
02
0.3
1.2
(a) Final scores in Ant
TD2 DCh
T32 w PCN
703 VCV
5:00
0.26
0.00
0.08
0.50
0.80
0.95
1.04
(b) AUC scores in Ant
1D3 wi DOt
T13 POR
0.00
0.15
0.24
0.32
0.05
0.12
0:38
0.24
0.00
0:36
0.44
0.32
0.72
0.00
0.30
Final scores over all MuloCo tasks
Medlan
Optimaliiyy
T03 wroch
TB3 wr
TD3 VCn
103
0.02
0.15
0.24
0.32
0.03
0.18
D:54
0.0
0.3
0.2
0.5
0.7
0.8
0.8
1.0
AUC all MuloCo tasks
Figure 17: Reliable (Agarwal et al. 2021 metrics for TD3 with different churn reduction options.
Med
ICH
Mean
Optimallyy
340 HOK
SAC VER.
SAC-
2.4
0.5
0.8
0.4
0.6
0.5
1.0
044
0.5
0.8
0.2
0.4
0.6
(a) Final scores in Ant
Median
Mean:
Octimallty
SAC
540 wi HOK.
S40 VOK
SAC:
0.25
0.50
0.75
1.00
0'3
5.6
0.
0.25
0.30
0.75
1.00
0.25
0.50
0.75
(b) AUC scores in Ant
SAC DCR
SAC wi PCR
SAC w/VCR
0.3
674
0.5
5.30
8.55
D-
055
055
0.5
0.7
Final all MuJoCo tasks
Mesdan
SAC oCB.
S6C PCR
SAC VCn
SAC
03
5.4
0.5
0.6
0.3
0.4
0.3 2.5
0.6
0.4
0.5
0.5
0.6
0.7
AUC all MuJoCo tasks
Figure 18: Reliable Agarwal et al.
2021 metrics for SAC with different churn reduction options.
26

---

D.4 More Results of CHAIN for IQL with Sequential Training
As mentioned in Section 6.3 the policy network of IQL has no impact on the training of the value networks, since the value networks (i.e., Q and V) are trained purely based on in-sample data without accessing a' = To (s"). Thus, although the policy and value networks of IQL still have churns, the chain effect of churn does not apply in this case.
The default implementation of IQL follows the fashion of iterative training between the policy network and the value network(s). Since the training of the value networks of IQL is independent of the policy, one natural idea is to first fully train the value networks for a sufficient budget and then train the policy value with the fashion as sequential training of IQL.
and frozen value networks. We call this actor-trained-
We slightly modified the training process of the CORL implementation of IQL to realize the sequential training of IQL: (1) First train the value network and Q network for 1 M steps; (2) Then train the policy network for IM steps with the value network and Q network frozen; (3) We do not modify any other implementation detail and use the same hyperparameters: (4) We check the learning curves of the policy network and the final scores. We call this variation IQL (sequential), The total number of gradient step is the same as the default IQL implementation where the critic and actor are trained iteratively.
We report the final scores of IQL (sequential) with means and standard errors over 12 seeds in Table g The results show that IQL (sequential) performs worse than IQL in 5 of 6 tasks. The difference between IQI. (sequential) and IQL. can fully attributed to the difference in the training dynamies, mainly on the policy network. This also means exposing the actor network to the training process of the Q network is helpful compared to training the actor based on a sufficiently trained and frozen Q network. To provide some possible explanation, a product of accumulated value churns, and the difference in the training dynamics of the policy network of IQL results in further difference in the policy output it is somewhat tricky to explain the difference between IQL (sequential) and IQL. The dynamics is beyond the scope of the chain effect of churn mechanism studied in our work.
guess that this is because the final Q network
oft-olesample states. However,
Table 8: Results for IQL, IQL (sequential) and CHAIN IQL in Antmaze.
Task
IQL
IQL (sequential)
CHAIN IQL (PCR)
CHAIN IQL (VCR)
AM-umaze-v2
77.00 5.52
60.00 + 3.91
84.44 + 3.19
83.33 + 2.72
AM-umaze-diverse-v22
54.25 + 5.54
55.00 5.46
62.50 3.75
71.67 + 7,23
M-mediuum-pla--22
65.75 + 11.71
52.50 3.36
72.50 2.92
70.00 :: 3.33
mmciium-diverse-..
73.75 5.45
53.33 + 5.93
76.67 + 4.51
66.67 + 3,79
AM-large-play-v2
42.00 + 4.53
17.5 + 4.10
50.00 + 4.56
43.33 + 4.14
AM-large-diverse-22
30.25 1 3.63
5.83 1 2.19
26.67 3.96
31.67 1: 2.31
D.5 More Results for Scaling PPO with CHAIN
We take PPO and MuJoCo tasks as the exemplary setting and widen both the policy and value networks by scale-up ratio within {2, 4, 8, 16). Note that the default network architecture (i.e., when the scale-up ratio equals one) for both the policy and value networks is two-layer MLP with 256 neurons for each layer, followed by an output layer.
Inspired by the prior study [hhando-Ceron et al., 2024), in addition to directly scaling PPO up ('direct'), we use two variants that use different learning rate settings for comparison: (1) 'linear' means using a decreased learning rate as ir scale- ratio, and (2) 'sqrt' means using ir / sqrt (scale-up ratio). The results of scaling PPO with different learning rate settings are shown in Figure 19 by different colors.
As expected, observed that the performance of PPO degrades as the increase of the scale-up ratio severely. We found using a decreased learning rate with 'linear' or 'sqrt' alleviates the degradation of PPO scaling some degree.
The evaluation results of the effect of CHAIN this sealing setting are shown in Figure 7 Figure ??
and Table 3 discussed in the main body of this paper.
27

---

Walker2d-v4 (2M)
Ant-v4 (2M)
Humandid-v4 (2M)
2500
6000
2500
Performance (score)
2000
Performance (scone)
1500
2000
Performance
2300
3000
10000
dircct
sort
4
16
16
Scale Up Rati
Scale Up Ratio
Scale Up Ratio
Hopperv4
HalfCheetah-v4
2400
2200
Performance (score)
2300
1800
2000
1900
1800
1400
16
Up
Up Ratio
Figure 19: The results for PPO scaling by widening with different learning rate settings. 'direct' means using the default learning rate, 'linear' means using a decreased learning rate as ir scale-up ratio, and 'sqrt' means using 1r sqrt (scale-u ratio).
Antov4 (2M)
Humanold (2M)
Walker2d (2M)
3000
2500
4090
Poromancc (core)
2000
1.00
8000
1000
ow/sort (P)
500
CHAIN
CHAIN sort
2000
20
16
Scale Up
Scale
Scale Ratio
(a) PPO Scaling by Widening
Antivá (2M)
Humanoid (2M)
Walker (2M)
3500-
Peformette (ecorel
2000
2000
12200-
1500
1000
PPO
PPO (wy sprt
2000
CHAiN PPO
CHAIN PPO (w sortin
500
1500
16
Scale Up Raiio
Sea Up Raão
Scale Up Ratio
(b) PPO Scaling by Deepening
The results in terms of episode return for scaling PPO with CHAIN. CHAIN helps to scale almost across all the configurations.
Figure 20:
D.6 More Results for CHAIN v.s. Slowing Down Learning
Since churn accompanies each time mini-batch training performed for the networks, natural question is: whether slowing down learning can alleviate the issue of churn. In principle, using smaller learning rates target network replacement rates should lead to less churn. This is because churn is positively related to the parameter update amount (as shown by the NTK expressions in Eq. 2 and a slower target network also slows the chur that occurs instantly in each training more when computing the target value to fit for the critic-network.
Empirically, we ran DDQN in Asterix and Freeway and TD3 in Walker2d and Ant, with different learning rates and target network replacement rates. The results are summarized Table 9 We can observe that either reducing learning rate or target network replacement rate often leads to worse
28

---

performance, especially for TD3. To some extent, this also matches the common knowledge in the RL community.
Table 9: Different learning rates (Ir) and target network replacement rates (trr), e.g., 2/2" means "divided by 2". Mean final episode returns over six seeds are reported.
Alg. Task
Walker2d
Ant
Alg. Task
Asterix
Freeway
TD3
4059.45
3069.14
DDQN
15.05
49.21
TD3 (Ir 2)
2774.87
2408.75
DDQN (Ir 2)
19.26
55,83
TD3 (Ir/ 10)
1314.12
1023.78
DDQN (Ir / 10)
16.20
49.20
TD3 (trr 5)
3170.18 18
3375.33
DDQN (trr 55)
11.36
50.76
TD3 (trr 5)
4057.33
2743.25
DDQN (trr 5)
18.26
54.33
This indicates that the issue of churn cannot be addressed by reducing learning target network replacement rate (which usually slows down the learning process). Churn a "by-product" of the training of DRL agents and should be addressed separately.
29

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and introduction summarize contributions and are supported by our formal analysis in Section and our experiment results Section 6
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Appendix A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
30

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide sufficient experiment and implementation details in Section 6 and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
31

---

Answer: [Yes]
Justification:
Our code can be found at
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ //ippe.cc/pullic/guuii wwwiii/gguie///eeeeeeoecc CoodeSubiiii ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide necessary experimental details in Section 6 and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The results in our paper are accompanied by error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
32

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the details of compute resources Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https /tteee///eeuuuie http/nnurppp.p. http////ccctttticccciiieaalll.a.
Answer: [Yes]
Justification:
This work conforms with the NeurlPS Code Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper presents work whose goal is to advance the field of Deep Rein- forcement Learning (DRL). No specific real-world application concerned. Our study on the churn phenomenon may help us better understand and control the behaviours of DRL agents, which can have positive impact in general the applicability and safety of DRL techniques.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
33

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All the environments, data and codes of baseline methods used this paper are publicly available Github. We cited the original papers and provided the URLs the assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
34

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide the necessary details for implementing our proposed method in this paper. We will provide a README document alongside the code release after the review process.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
35

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
36


Reviews:
[{"review_id": "JTSAEYIQwl", "reviewer": "Reviewer_DNMp", "summary": "This paper focues on the training instability from the non-stationary nature of DRL, whose pheononmena are unexpected shift in policy and value network outputs (policy churn and value churn). To mitigate policy and value churn, the proposed CHAIN algorithm add penalize in the policy and value updates.", "questions": "1. CHAIN introuduces two extra coefficients, $\\lambda_Q$ and $\\lambda_\\pi$ to control the degree of regularization. I find that for different benchmark suites or tasks or backbone algorithms, are set differently. So I have the following questions:     \n\n   What is the insight behind the chosen hyperparameters for different settings? If the potential readers want to apply CHAIN to other settings, how can they set the two hyperparameters? Actually, I find some explanations on this in Appendix C.2, but it requires making [policy loss]/[regularization loss] within 0.1 to 0.01, thus it needs to first try and then observe the ratio and then turn, maybe need several runs, whose costs could not be ignored.  Introducing extra hyperparameters and a simple one set of them could not suffice to get satisfying results for varying settings is not encouraged, as actually, it may output worse performance than the backbone algorithm if setting wrong hyperparameters. \n\n2. How many seeds do you use for the experiments in your main paper? For example, Figure 6. The CHAIN SAC in Figure 6 does not show a statistically significant improvement to SAC in Hopper-v4, Walker2d-v4, and HalfCheetah-v4, which may bring some doubts on the effectiveness of CHAIN, especially considering that an unsuitable hyperparameters set may provide even worse results of CHAIN SAC.\n\n\n\n3. As the paper focuses on mitigating the training instability of DRL and the non-stationary properties, some other methods may need to be considered to compare to or discuss, such as AlgaeDice [1], etc. \n\n\n\n4. Further, in online RL, over-conservative may hinder performance, so maybe the authors need to discuss well: whether the CHAIN would negatively influence the exploration in online RL and then hinder the asymptotic performance.\n\n5. And, another style of work, for example, [2],  in online RL actually encourages the network neurons to awaken, thus preventing the local optimal. Could you provide some explanations on whether CHAIN would deteriorate the local optimality issue?\n\n[1] Nachum, O., Dai, B., Kostrikov, I., Chow, Y., Li, L., & Schuurmans, D. (2019). Algaedice: Policy gradient from arbitrary experience. *arXiv preprint arXiv:1912.02074*.\n\n[2] Xu, Guowei, et al. \"Drm: Mastering visual reinforcement learning through dormant ratio minimization.\" *arXiv preprint arXiv:2310.19668* (2023).", "limitations": "The authors have acknowledged the limitations.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "CHAIN is a versatile solution that can be integrated into different backbone algorithms.", "weaknesses": "* Extra introduced hyperparameters and no-trivial hyperparameters setting. \n* No statistically significant improvements in many tasks, even equipped with good hyperparameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s valuable review and the recognition of the versatility of the proposed method. Our method is simple but versatile to improve the performance of many learning algorithms. We show that the method works across continuous and discrete control tasks, online and offline settings. Moreover, in our additional results (Table 9 in the one-page pdf) we show that **the method can greatly improve the scaling of RL algorithms**. This scaling is also improved by an adaptive version of our method that reduces hyperparameter tuning.\n\n\nThe main concerns focus on the hyperparameter choice for different settings and CHAIN’s effect on exploration and optimality. Our response aims to clarify these aspects in detail.\n\n\n> Q1: The insight behind the chosen hyperparameters for different settings \n\n\nHow to pick or adjust the regularization coefficient is an unavoidable tricky problem for most regularization-based methods. In the context of our work, the difference in the scale of the quantities like Q value and policy objective across different settings leads to the use of difference coefficients. This is also a long-standing problem of DRL.\n\n\nTo relieve the pain of manually picking coefficients in different tasks and settings, we additionally propose **a simple but effective method for automatic adjustment of** $\\lambda_{\\pi}, \\lambda_{Q}$ with a target relative loss scale denoted by $\\alpha$. It is realized by maintaining the running means of the absolute policy ($| \\bar L_{\\pi}|$) or Q loss and the PCR ($| \\bar  L_{PC} | $) or VCR term, and dynamically computing the regularization coefficient, e.g., $\\lambda_{\\pi} = \\frac{\\alpha | \\bar L_{\\pi}| }{| \\bar  L_{PC} | }$. This method adjusts the coefficient dynamically to keep the consistent relative loss scale $\\alpha$.\n\nOur additional experiments in the one-page PDF show that **this method matches or surpasses the results achieved by the manual choices of** $\\lambda_{\\pi}, \\lambda_{Q}$ for DDQN in five MinAtar task (Figure 16), for PPO in five MuJoCo tasks (Figure 17) and ten additional DMC task (Figure 19), and for TD3 in four MuJoCo tasks (Figure 17).\n\n> Q2: Whether the CHAIN would negatively influence the exploration in online RL and then hinder the asymptotic performance\n\n\nIn principle, reducing churn suppresses the correlation of network outputs among different data. This encourages the agent to keep the independence of the action on different states. Thus, we hypothesize that it could help to **keep the diversity of action** (in other words, prevent the collapse of action mode as often seen in the cases with severe learning instability for both RL[2] and LLM [3]) and **positively influence exploration**. \n\nIn our experiments, we can see that CHAIN **improves the asymptotic performance in the tasks that require effective exploration** even though the value or policy churn is significantly reduced. For MinAtar, the original MinAtar paper mentioned “Seaquest and Freeway present a greater exploration challenge than the other games” on page 5, and our results in Figure 4 show CHAIN improves the final performance of DDQN by clear margins in these two environments.\n\nMoreover, we additionally provide the comparison between CHAIN PPO and PPO in two sparse-reward tasks of DeepMind Control Suite (DMC): ball_in_cup-catch-v0 and cartpole-swingup_sparse-v0. The results are shown in the first two subplots in Figure 19. We can see that CHAIN PPO achieves a higher asymptotic performance in ball_in_cup-catch-v0 and achieves almost the same performance in cartpole-swingup_sparse-v0.\n\n\n> Q3: DrM [2] encourages the network neurons to awaken, thus preventing the local optimal. Whether CHAIN would deteriorate the local optimality issue\n\nDrM prevents the neurons from being dormant with an adaptive perturbation reset method. This addresses the loss of plasticity/learnability and thus alleviates the local optimality issue.\n\nIn principle, CHAIN has **a positive effect in preventing the loss of plasticity**. As in Equation 3, reducing churn encourages $k_{\\theta}, k_{\\phi}$ to 0. This prevents the empirical NTK matrix from being low-rank, which is shown to be a consistent indicator of plasticity loss by [1] (they also claimed empirical NTK is a more reliable indicator than dormant neuron ratio).\n\n\nIn our experiments, we can see that CHAIN improves the final scores in most cases across MinAtar, MuJoCo (especially with CHAIN PPO) and additional ten DMC tasks (in Figure 18 of the one-page pdf).\nAdditionally, we provide more results in **the one-page pdf** to show that CHAIN improves the final performance both when **running longer** (Figure 17 Ant and Humanoid for 10e6 steps) or with **larger networks** (Table 9). \n\n\n> Q4: The discussion on AlgaeDice\n\n\nWe appreciate the reviewer for reminding us of this. AlgaeDICE aims to address the limitation of on-policy samples by re-expressing the on-policy max-return objective by an expectation over any arbitrary off-policy data distribution, which is followed by a solution in the form of dual optimization. \n\nAlgaeDICE does not have a direct relation to churn or generalization of DRL agents, however, we believe it is orthogonal to CHAIN in mitigating training instability. We have integrated AlgaeDICE and more in our related work discussion.\n\n> Q5: The number of random seed\n\nIn the one-page pdf, we provide the results across 12 seeds for CHAIN PPO in Figure 17 and \nadditional results for ten DeepMind Control Suite tasks with 12 seeds in Figure 19. The results demonstrate the effectiveness of CHAIN in improving the learning performance of PPO.\n\nWe found that 6 seeds are sufficient to present a clear evaluation for DDQN in MinAtar. We will run more seeds for TD3/SAC as suggested.\n\n---\nReference:\n\n[1] Disentangling the causes of plasticity loss in neural networks. 2024. \n\n[2] Overcoming Policy Collapse in Deep Reinforcement Learning. 2023.\n\n[3] Controlling Large Language Model Agents with Entropic Activation Steering. 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zZAWFGJqoq", "reviewer": "Reviewer_Qjzk", "summary": "Deep RL optimisation exhibits many instabilities and performance collapses. Schaul et al. [2022] discuss a pathology termed the \"policy churn\", where even a single update to the value network frequently changes the optimal action for a huge fraction of all states (most of which were not present in the training batch). Following in these steps, the current paper characterises churn through the lens of generalised policy iteration, and proposes a simple regularisation loss that can be intuitively described as restricting the actor and critic predictions from changing \"too much\" on other states.", "questions": "**IQL**: Line `338` says you apply CHAIN to IQL --- could you explain how? as far as I can tell, IQL does not need to train an actor network in tandem with a critic. It's likely that I did not fully understand this part.", "limitations": "The authors do mention the limitations of the work, but the paper could be greatly improved by reducing the role of the mathematical exposition that takes up the majority of the paper, and giving more space to the experiments.\n\nThis is an important area of work that can benefit from new empirical findings, and I am quite willing to improve my rating if my concerns are addressed.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The authors argue convincingly that churn effects in actor and critic optimisation can interact amplify each other, in what they call the \"chain effect\", and propose a surprisingly simple regularisation loss to weaken this effect. As a result, the churn measurably goes down, which appears to correlate with better learning performance.\nAs there are few papers on mitigating churn, I think these findings are useful and can help inform further research.", "weaknesses": "# Writing\nI have some concerns about the writing. The topic is important, and relatively new, so good communication would be valuable.\n\nThe paper is quite well written until page 4, but later, it gets hard to follow. \nFor example, the language starts to get more vague, e.g. line `176` -\n> One thing to note is the two types of deviation are derived in an **exchange manner** between the value and policy churn,\n\nI found Section `3.3` particularly hard to follow and had to reread it many times. It introduces a lot of new symbols in one go, uses them in 1/3rd of the page, and then moves on to the next topic. The section's conclusion is that churn and parameter update bias amplify each other as the optimization goes on (something which the reader is already informally aware of since early on in the paper).\n\nAfterwards, the main method (CHAIN) is introduced as late as *page 7*, following which the paper appears quite rushed.\n\nI think the paper could benefit from:\n- moving the NTK discussion (which is never used afterwards) and everything theoretical after that to the appendix. The math may be saying the right things, but I did not find it *useful* in this context.\n- focusing on describing CHAIN and its experiments in more detail.\n\n`326`: You mention Fig 5, but I think you mean Fig 6\n\n`328`: I'm not sure what the following bolded part means. Does it mean that the target critic network slows down credit assignment / weight updates, and therefore reduces churn? It would be helpful to be more verbose in the text right here. \n> We hypothesize that this is because policy interacts with the environment directly **and the target critic network also helps to cap the value churn**. \n\n\n# Experiments\n\nThe churn reduction looks quite noticeable and the MinAtar evaluations also show a notable performance boost. How much of this observed churn reduction can be explained by simply \"slowing down\" the parameters' changes? \n\nI think this is quite important to test, one could also simply train both actor and critic with:\n1. slightly reduced learning rates\n2. different target network momentums\n... and observe if the churn also reduces due to those things (and whether the agent performance also correspondingly improves). I believe such an experiment would give clearer insights, and possibly strengthen the paper.\n\n\nLine `624`: \n>  Between TD3 and SAC, CHAIN-PCR works better for TD3 rather than SAC. We hypothesize that this is because KL-based PCR used for SAC poses more challenges in code implementation.\n\nThat is concerning, because it sounds like a bug in your code. Not affecting the entire paper, but it does make it hard to believe the SAC results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s valuable review and the recognition of the value of the chain effect and the method proposed in our work. Our additional results in the one-page pdf further demonstrate **the effectiveness of CHAIN in ten DMC tasks and improving the learning when larger networks are used**. We also believe that our study on the chain effect of churn can inspire more future work on understanding and addressing the learning issues of DRL.\n\nThe main concerns focus on our writing and several experimental details. Our response aims to systematically clarify these aspects.\n\n> Q1: Churn reduction v.s., Slowing down the parameters’ changes (with reduced learning rates and different target network momentums)\n\n\nWe appreciate the reviewer for pointing out this insightful investigation. We ran additional results for using different learning rates and target network replacement rates. The results are summarized **in Table 10 of the one-page pdf**.\n\nIn terms of learning performance (i.e., final episode return), we can observe that **either reducing learning rate or target network replacement rate often leads to worse performance**, especially for TD3. To some degree, this matches the common knowledge in the RL community.\n\nAs to churn reduction, in principle, using smaller learning rates or target network replacement rates should lead to less churn. This is because churn is positively related to the parameter update amount (as shown by the NTK in Equation 3) and a slower target network also slows the churn that occurs instantly in each training more when computing the target value to fit for the critic network. We also observed empirical evidence for this (omitted due to one-page space limitation).\n\nThis indicates that **the issue of churn cannot be addressed by reducing learning rate or target network replacement rate** (which just slows down the learning process). Churn is a “by-product” of the training of DRL agents and should be addressed separately. Actually, we observed that applying CHAIN when using smaller learning rates also improves the learning performance in some tasks, which is omitted due to one-page space limitation.\n\n\n\n> Q2: “Line 338 says you apply CHAIN to IQL --- could you explain how? as far as I can tell, IQL does not need to train an actor network in tandem with a critic. It's likely that I did not fully understand this part”\n\nAccording to Section 4.3, Equation 7 ($L_{\\pi}(\\phi)$) and Algorithm 1 in the original paper of IQL (“Offline Reinforcement Learning with Implicit Q-Learning”), **a policy network is explicitly trained** with advantage-weighted regression based on the Q and V networks. Concretely, a Gaussian policy is generated by the policy network of IQL for action selection as the D4RL tasks in our experiments have continuous action spaces. \n\nTherefore, we apply CHAIN to IQL by optimizing $L_{\\pi}(\\phi)$ together with $L_{PC}(\\phi, B_{ref})$ as Equation 10 in our paper. Please let us know if there are any further questions about this point.\n\n\n> Q3: On the writing suggestions\n\nWe appreciate the reviewer for pointing out these issues in our writing and providing suggestions, which are very valuable to the further polish of our paper. \n\n- [Line 176, “exchange manner”] We amended this improper expression by re-rewriting the sentence to “the two types of deviation show the interplay between the value and policy churn as the value churn causes the deviation in policy (i.e., the action gradient) and the policy churn causes the deviation in value (i.e., the policy value)”.\n\n- [Section 3.3 and the organization of NTK discussions and experiments] In Section 3, we aim to present (1) how parameter update causes churn with the NTK in Equation 3, (2) the interplay between the policy and value churn with the NTK in Equation 5, (3) how churn introduces bias in parameter update with Equation 6. The three components form the cycle: parameter update —> churn —> parameter update —> …. Therefore, we are afraid that completely moving the NTKs to the appendix could make some readers lose track of these connections which causes instability.\nHowever, we agree with the reviewer that we can simplify and shorten the content of Section 3 to make the thread clearer and more prominent, and then make more room to strengthen our experiment section with more analysis and our additional results (as provided in the one-page pdf). We have **re-organized the content in our revision and included additional experimental analysis** on scaling networks as well as an updated version of CHAIN with the method for adaptive regularization coefficients to reduce hyperparameter tuning\n\n\n- [Line 326] We amended it to be Figure 6 as pointed out.\n\n- [Line 328, “the target critic network also helps to cap the value churn”] It means that the value churn that accompanies each training of the critic network is not immediately (or fully) reflected in the target critic instantly due to delayed synchronization of the target network. Thus, the target critic should reduce the value churn by a factor associated with the exponential moving coefficient $\\tau$ (i.e., the momentum hyperparameter). We added these explanations to make it clearer.\n\n- [Line 624, “KL-based PCR for SAC”] We meant to express that the variation (regarding the scale and range) is higher in optimizing the MaxEnt objective and KL-based PCR term together for SAC than in optimizing the Q objective and L2-based PCR term for TD3. Another hypothesis is that the MaxEnt nature of SAC prefers the encouragement of more stochasticity in policy. We re-wrote the sentence to eliminate the expression issue.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aJBUhGiEh2", "reviewer": "Reviewer_c7UJ", "summary": "The authors focus on improving the current state of deep reinforcement learning by addressing the churn effect in deep neural network training. Churn effect in deep reinforcement learning is the phenomenon where output predictions for data outside training dataset can change during training updates. This can lead to instability, suboptimality and even collapse. Previous works in addressing churn are limited in scope and do not provide a method to counter churn effect. In this work, the authors perform a detailed exploration of churn effect in RL and propose an algorithm to counter this issue. They characterize churn in a view of Generalized Policy Iteration with function approximation and discover a chain effect of churn. This chain effect leads to a cycle that compound and bias the learning dynamics throughout the iteration. The authors study the effect of churn across different setups and propose CHAIN algorithm, which can be easily plugged into most existing DRL algorithms. Experiments show that their method is effective in both reducing churn and improving learning performance across online and offline, value-based and policy-based RL settings.", "questions": "Please see weakness section", "limitations": "Yes, the authors mention limitations in the paper.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The problem is well motivated, the authors provide detailed explanations of what the problem is, why its important, what has been done previously, and a detailed explanation of their methodology.", "weaknesses": "1.\tIn the introduction, please provide a short intuition behind the CHAIN algorithm, how is it achieving the explicit control of churn. Perhaps mention that by modifying the original loss function, CHAIN can achieve this.\n2.\tExpand on the figure captions. This applies to all figures 1-6. \n3.\tFor figure 1, what is meant by the policy churn and value churn arrows? Are there any updates going on in between? In the description, the authors mentioned “explicit training”, please elaborate what is meant by this.\n4.\tFor figure 2, expand the caption for 2b, in the figure 2a there is an arrow without direction, please double check if this is accurate, if yes mention in the caption why that is the case\n5.\tFor figure 3-6, mention the key takeaways from the graphs in figure caption\n6.\tIn table 1, CHAIN seems to perform worse for AM-large-diverse-v2 dataset, any explanation/intuition as to why that may be the case?\n7.\tFigure 3 appears after figure 4, please correct this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s careful review and constructive feedback, and the reviewer’s recognition of the motivation of this work and the importance of studying the churn problem in deep RL. Our additional results in the one-page pdf **further demonstrate the effectiveness of CHAIN** equipped with our method for **auto-adjustment of the regularization coefficient**, with **ten new DMC tasks** and **an additional investigation on learning with larger networks**.\n\nThe main concerns focus on several expression details and the reviewer also noted valuable writing suggestions. Our response aims to clarify these aspects in detail.\n\n> Q1: “For figure 1, what is meant by the policy churn and value churn arrows? Are there any updates going on in between? In the description, the authors mentioned “explicit training”, please elaborate what is meant by this.”\n\nThere are no updates between $Q_{\\theta}$ and $\\tilde{Q}\\_{\\theta}$ (and  $\\pi_{\\phi}$ and $\\tilde{\\pi}\\_{\\phi}$), which are denoted by the two red wave arrows. The only “explicit training” is the policy evaluation and the policy improvement, which are denoted by the black arrows.\n\nEach function approximation training accompanies churn, i.e., the network is changed by the (active) explicit training and the (passive) churn. Different from the conventional view that takes the two components as a whole, in Figure 1, we use **a two-step view** for both the value and policy learning, e.g., for the value learning case, step 1 is the value network is updated to $Q_{\\theta}$ with explicit training on a batch of data for the policy evaluation, and step 2 is the value churn occurs and leads to $\\tilde{Q}_{\\theta}$.\n\nIn practice, the two steps happen at the same time. Thus, $Q_{\\theta}$ and $\\pi_{\\phi}$ in Figure 1 are **the virtual intermediate states** in the learning process, used for the need of a disentangled illustration.\n\nWe appreciate the reviewer for pointing out this. We have clarified it and amended the notations to eliminate the confusion in the revision.\n\n\n\n> Q2: “In table 1, CHAIN seems to perform worse for AM-large-diverse-v2 dataset, any explanation/intuition as to why that may be the case?”\n\nAfter adding the seed number to 12 and trying smaller regularization coefficients $\\lambda_{\\pi}$, the scores of CHAIN IQL for AM-large-diverse-v2 are: \n26.67 $\\pm$ 3.96 for $\\lambda_{\\pi}=1000$ (the one reported in Table 1), 28.33 $\\pm$ 4.05 for $\\lambda_{\\pi}=500$,  35.0 $\\pm$ 4.48 for $\\lambda_{\\pi}=200$, and 32.5 $\\pm$ 2.08 for $\\lambda_{\\pi}=100$.\n\nThus, we hypothesize that $\\lambda_{\\pi}=1000$ slightly over-regularizes IQL while using $\\lambda_{\\pi}=200$ or $100$ improves IQL. This indicates that there is still room to achieve a better score with a smarter strategy for coefficient choice.\n\n\n\n\n> Q3: On the writing suggestions\n\nWe appreciate the reviewer’s valuable suggestions on improving the expressions in the introduction, the figure captions, etc.\n\n- [Introduction, “please provide a short intuition behind the CHAIN algorithm, how is it achieving the explicit control of churn. Perhaps mention that by modifying the original loss function, CHAIN can achieve this”] \nWe added the details “The main idea of CHAIN is to reduce the undesirable changes to the policy and value networks for states (and actions) that are outside of the current batch of data. The motivation is similar to the monotonic improvement principle of PPO and TRPO, which in their cases is achieved by improving just the action distribution for the samples in the current batch. Concretely, CHAIN minimizes one additional churn reduction regularization term computed with a separate data batch along with the optimization of the original policy or value learning objectives” to describe the intuition, right after the first sentence in Line 50.\n\n- [Figure 2a, “there is an arrow without direction”] It should be an arrow towards the right (i.e., from $Q_{\\theta_t}$ to $\\pi_{\\phi_{t+1}}$) and we have amended it. \n\n- [Figure 3-6] We have added takeaways in the captions as suggested.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zIAwgLDkDR", "reviewer": "Reviewer_oLah", "summary": "This work studies a phenomenon called churn, which refers to that the outputs of a network after updates could change to unexpected values for input data not included in the training batch.\nSpecifically, it studies the value churn, the policy churn, and the interplay between them in deep reinforcement learning (DRL) setting.\nThen a simple regularization method is propose to reduce the the value and policy churn. Empirical results demonstrate the effectiveness of the proposed solution across many DRL settings, such as value-based methods, policy gradients methods, and offline RL.", "questions": "How does policy/value churn related to catastrophic forgetting? For example, Figure 2 in MeDQN paper[8] shows that a change of greedy action could result in forgetting. Will reducing forgetting reduce churn? Or vice versa?", "limitations": "Yes, the limitations are discussed in the work.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Except some minor issues, the paper is generally well-written and easy to follow.\n- The proposed method is simple and effective, verified in various RL settings.\n- Besides improving the final performance, direct evidences are also provided to show that the proposed method indeed can reduce churns.", "weaknesses": "## Originality\n\nAlthough this work studies a relatively new concept (churn), the investigations of the root causes of this phenomenon, the proposed method, and the idea behind are not novel.\n  - The churn phenomenon is strongly related to interference and generalization of neural networks. And using NTK to measure the interference and generalization is already proposed by previous works for DRL, such as [1-4]. These works should be included in the related works and discussed. Out of DRL, similar phenomena are also studied under supervised learning setting, such as [5-7].\n  - I like the simplicity and effectiveness of the proposed algorithm (CHAIN), but it is really similar to MeDQN(R)[8] although the latter work focuses on reducing forgetting instead. This greatly damages the novelty of this work unfortunately.\n\n\n## Quality\n\nThe empirical and theoretical results of this work can be further improved.\n  - Equation 1 is inappropriate as the definition of churn. An absolute operation or square operation should be applied. Otherwise, positive and negative values may cancel out. The issue exists for Equation 4 as well. This is a serious issue since changing Equation 1 would also results in changing the latter derivations in the paper. For example, Equation 5 may no longer hold.\n  - In Equation 3, although using $B_{train} = \\\\{s,a\\\\}$ and $B_{ref} = \\\\{\\bar{s}, \\bar{a}\\\\}$ help with notation clarity. However, it may lead to misleading derivations. For example, the cancel-out effect is no longer revealed (see previous point about Equation 1). Given this, I would suggest to consider the general case of using more samples in $B_{train}$ and $B_{ref}$.\n  - All experiments use only 6 random seeds which are not enough. I would suggest at least 10 seeds.\n  - The theoretical results are interesting and useful to help understand and support the intuitions behind the problem. However, I don't see how they could help with the proposed algorithm. In other words, the connection between theoretical derivations and the proposed method is too weak.\n\n## Clarity\n\nThere are some minor issues.\n  - What is the difference between $Q\\_{\\theta}$ and $\\tilde{Q}\\_{\\theta}$ ($\\pi\\_{\\phi}$ and $\\tilde{\\pi}\\_{\\phi}$)? I understand that $Q_{\\theta}$ is an approximation of $q^{\\pi}$ (Section 2.2) where $\\theta$ are the parameters of the Q-network. But what is $\\tilde{Q}\\_{\\theta}$ in Section 3.1 and Figure 1 exactly? Isn't it also an approximation of $q^{\\pi}$? In fact, I would suggest to remove $\\tilde{Q}\\_{\\theta}$ and $\\tilde{\\pi}\\_{\\phi}$ to simplify notations, as they seem to be redundant (but I might be wrong).\n  - Missing one right parenthesis in the definition of $\\delta(s,a)$ in Line 110.\n  - Typo in Line 322: policy churn reduction (VCR) --> value churn reduction (VCR).\n\n\n1. Liu, Vincent, et al. \"Towards a practical measure of interference for reinforcement learning.\" arXiv preprint arXiv:2007.03807 (2020).\n2. Liu, Vincent, et al. \"Measuring and mitigating interference in reinforcement learning.\" Conference on Lifelong Learning Agents. PMLR, 2023.\n3. Joshua Achiam, Ethan Knight, and Pieter Abbeel. Towards characterizing divergence in deep Q-learning. arXiv:1903.08894, 2019.\n4. Emmanuel Bengio, Joelle Pineau, and Doina Precup. Interference and generalization in temporal difference learning. arXiv preprint arXiv:2003.06350, 2020.\n5. Fort, Stanislav, et al. \"Stiffness: A new perspective on generalization in neural networks.\" arXiv preprint arXiv:1901.09491 (2019).\n6. He, Hangfeng, and Weijie Su. \"The Local Elasticity of Neural Networks.\" International Conference on Learning Representations.\n7. Lan, Qingfeng, and A. Rupam Mahmood. \"Elephant neural networks: Born to be a continual learner.\" arXiv preprint arXiv:2310.01365 (2023).\n8. Lan, Qingfeng, et al. \"Memory-efficient Reinforcement Learning with Value-based Knowledge Consolidation.\" Transactions on Machine Learning Research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s careful review and the recognition of our experimental work. Our additional results in the one-page pdf further demonstrate the effectiveness of CHAIN in **ten more DMC tasks** and **improving the learning when larger networks are used**.\n\nThe main concerns focus on the relationship between our work and the previous works mentioned by the reviewer on catastrophic forgetting and interference. Our response aims to systematically clarify these aspects.\n\n> Q1: CHAIN v.s. MeDQN(R) [8]\n\nCHAIN and MeDQN(R) are **different on two key points**. \n\nMeDQN(R) was created to improve the DQN algorithm only when there is a target network available to use as a fixed point to reduce forgetting. **CHAIN does not need this fixed point** and CHAIN applies to both value-based and policy-based RL settings in a plug-in manner. \n\nThe second point is that **CHAIN does not aim to reduce forgetting or stop all types of churn**. Some churn is beneficial and we don’t want to over-constrain the algorithm otherwise we may lose the plasticity of the network. We show the importance of this point in our additional experiments in Table 9 of the one-page pdf, where the results show how **CHAIN improves the training of larger networks**. \n\nMoreover, in the next question, we discuss the distinction between CHAIN and MeDQN(R) regarding their effects in reducing churn and forgetting.\n\n> Q2: “How does policy/value churn related to catastrophic forgetting? For example, Figure 2 in MeDQN paper[8] shows that a change of greedy action could result in forgetting. Will reducing forgetting reduce churn? Or vice versa?”\n\nChurn is a natural behavior of neural networks. It accompanies each training and occurs instantly. To some extent, we consider that **catastrophic forgetting can be viewed as a consequence of the accumulation of churn**.\n\nTherefore, **reducing churn also helps to reduce forgetting** as it suppresses the accumulation. However, **reducing forgetting does not necessarily reduce churn** and **could even increase churn**. It depends on the method considered. \n\nConcretely, for example, MeDQN [8] reduces forgetting by preventing DQN’s value prediction from deviating from a previous copy (in practice, the target network). Note that there is a **time gap** between the current Q-network and the target network. Therefore, MeDQN does not reduce the churn (which occurs instantly) and could **incur extra churn when rolling back the network outputs to the target network**.\n\n> Q3: The relationship between churn and interference\n\nFor the earliest NTK-related paper in DRL [3] mentioned by the reviewer, we **cited it in Line 159 of our submission** (i.e., the first time we mentioned NTK in our paper), and our analysis with NTK expressions is inspired by it. Also following [3], we found [1,2,4] studied the interference as the change in value approximation error with different forms (e.g., squared TD/TD($\\lambda$) error), and they mainly focused on value-based RL (e.g., DQN).\n\nTo establish a possible view of the relationship between churn and interference: churn is the change in the network output, while inference is the change in the objective function of the network. Thus, we agree that the studies on churn and interference are closely related essentially.\n\n**We emphasize** that we also **study the churn in the policy network** (thus going beyond value-based RL). Moreover, we present **the interplay** between the value churn and the policy churn and the dynamics under **the chain effect**. This is **not studied in previous work** to the best of our knowledge.\n\nWe appreciate the reviewer also noted [5-7]. We included them in our revision to strengthen our related work section.\n\n> Q4: The definition of churn in Equation 1 (an absolute operation or square operation should be applied) and amendment of the derivations\n\nWe appreciate the reviewer for pointing out this discrepancy in our derivations. After carefully re-checking the derivation, we found this issue is amendable with several modifications (as shown below), and **the insights given by our derivations do not change**.\n\n**The amendment has been done as follows** (please **refer to the official comment attached for a complete description with math**):\n- First, we add the (signed) definitions of churn on a single data point, denoted by: $c_{Q}(\\theta, \\theta^{\\prime}, \\bar s, \\bar a), c_{\\pi}(\\phi, \\phi^{\\prime}, \\bar s)$.\n- Next, we re-write our Eq.1 as the mean of the (element-wise) absolute churn of each data point, i.e., with $| c_{Q}|,  |c_{\\pi}|$.\n- We then apply the same logic and amend the definition in Equation 4 by defining $d_{\\nabla_a}^Q, d_{Q}^{\\pi}$ and then using them to re-write $\\mathcal{D}\\_{\\nabla_a}^Q, \\mathcal{D}\\_{Q}^{\\pi}$ with absolute operations.\n- By using the absolute operation, there is no cancel-out issue any longer. This also makes it rationale to use $B_{\\text{train}}=\\\\{s, a\\\\}, B_{\\text{ref}} = \\\\{\\bar s, \\bar a\\\\}$ for simplification in our NTK discussions. We replace the $\\mathcal{C}\\_{Q}, \\mathcal{C}\\_{\\pi}, \\mathcal{D}\\_{\\nabla_a}^Q, \\mathcal{D}\\_{Q}^{\\pi}$ in Equation 3 and 5 with $c_{Q}, c_{\\pi}, d_{\\nabla_a}^Q, d_{Q}^{\\pi}$.\n- This logic also applies to the analysis in Equation 6 and further the analysis of the upper bound.\n\n> Q5: The connection between theoretical derivations and the proposed method\n\nThe main message of our formal analysis is the chain effect of churn in Figure 2. It directly motivates us to reduce the value or policy churn to suppress the chain effect. Then the concrete consequences presented in Section 4.1 guide us to the specific choices in different problem settings.\n\n> Q6: The difference between $Q_{\\theta}$ and $\\tilde{Q}\\_{\\theta}$ ( $\\pi_{\\phi}$ and $\\tilde{\\pi}\\_{\\phi}$) in Section 3.1 and Figure 1\n\nPlease kindly refer to **Additional Response** in the official comment attached.\n\n> Q7: Number of random seeds\n\nPlease kindly refer to **Additional Response** in the official comment attached.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate all the reviewers’ careful review and valuable comments. Here **we summarize the main points of our response to each review** and **the content of our additional results** enclosed in the one-page pdf.

&nbsp;

The summary of the responses to individual reviews:
- **[Reviewer DNMP]** We appreciate the reviewer’s recognition of the versatility of the method proposed in our work. The main concerns focus on the insight behind hyperparameter choice for different settings, and CHAIN’s effect on exploration and optimality. To this end, we propose **a simple but effective method for automatic adjustment of the regularization coefficients** by developing the insight of keeping a consistent relative scale between the regular policy or value loss and the churn reduction term. Our additional results (Figure 16-19 in the one-page pdf) show that **this method can match or surpass the results achieved by manually picked coefficients** across different tasks in MinAtar, MuJoCo and additional tasks in DeepMind Control (DMC) suite (Figure 19). We believe that this provides a useful reference for practitioners to use CHAIN in a different problem. Besides, we discuss **how CHAIN can have positive effects in encouraging exploration and preventing plasticity loss in principle**, and show comparable or better asymptotic performance in four exploration tasks (including 2 in MinAtar and 2 in DMC).
- **[Reviewer Qjzk]** We appreciate the reviewer’s recognition of the value of the chain effect and the method proposed in our work. The main concerns focus on an insightful discussion on “the effect of reducing learning rate or target network replacement rate in reducing churn” and several expression details. In the individual response, we show **either reducing learning rate or target network replacement rate slows down learning and leads to worse performance** (with the additional results in Table 10 of the one-page pdf). We also note that **using smaller learning rates or target network replacement rates does not address the issue of churn**, although it has less churn in principle (which is explained by our NTK expressions in Equation 3).
- **[Reviewer c7UJ]** We appreciate the reviewer’s recognition of the motivation of this work and the importance of our study. The main concerns focus on several expression details and the reviewer also noted valuable writing suggestions. In the individual response, we provide detailed explanations and describe how we took the writing suggestions to polish our paper concretely.
- **[Reviewer oLah]** We appreciate the reviewer’s recognition of our experimental work. The main concerns focus on the relationship between our work and the previous works mentioned by the reviewer on catastrophic forgetting (especially MeDQN) and interference. Our response explains how **reducing churn also helps to reduce forgetting** but **reducing forgetting does not necessarily reduce churn and could even increase churn**. We also point out the essential connection between churn and interference, and the clear distinction of our work on the study of policy churn, the interplay, and the dynamics under the chain effect.

&nbsp;

In the one-page pdf, we provide additional results:
- **[A simple method for automatic adjustment of the regularization coefficient (refer to Q1 of Reviewer DNMp for the details)]** Figure 16-18 show the results of our auto-adjustment method for CHAIN DDQN/PPO/TD3. The results show that, this method can match or surpass the learning performance achieved by manually picked regularization coefficients across different tasks and different settings. We believe that this method can help to relieve the pain of coefficient choice in practice, and inspire the development of smarter methods.
- **[Additional evaluation in ten DMC tasks]** Figure 19 shows additional results of CHAIN PPO for 10 tasks in DeepMind Control Suite, with two sparse-reward tasks (the first two).
- **[CHAIN helps the scaling of PPO]** Table 9 shows our additional investigation on the effect of CHAIN when scaling DRL agents up. We take PPO as the exemplary setting and widen both the policy and value networks to 4 and 8 times. The results demonstrate that CHAIN helps to scale up PPO and achieves clear improvement in terms of episode return. Moreover, we found that **larger networks exhibit more churn**. We hypothesize that it is a possible reason for the notorious scaling issue of DRL agents, and **CHAIN helps the scaling by reducing churn effectively**.
- **[(To Reviewer Qjzk) Churn phenomenon and learning performance when using smaller learning rates and target network replacement rates]** Table 10 shows smaller learning rates and target network replacement rates often slow down learning and lead to worse performance.

&nbsp;

With these additional discussions and experimental results, **we emphasize** that **our formal study on the churn in both value and policy** (including the interplay, the chain effect, and the concrete issues in popular DRL settings), **the simple and general churn reduction method** (including CHAIN and the additional auto-adjustment method), and **the empirical discoveries** (including what the churn is like in popular DRL agents, and how they can benefit from churn reduction) **are novel** to the best of our knowledge.

We believe that our paper can inspire more future work on understanding and addressing the learning issues of DRL from angles like churn, interference, and generalization as mentioned by the reviewers. And the method proposed in this paper can be implemented very easily and adopted in broader scenarios, e.g., PPO-based alignment of LLMs.

&nbsp;

Finally, we sincerely hope that our response can address the questions and concerns. We also hope that the reviewers can re-evaluate the value of our work based on the responses and the additional results. 
We are also more than willing to address any further questions or concerns during the discussion stage.

Meta-review: This paper studies a problem recently observed in the training dynamics of deep networks in reinforcement learning, aka churn. It refers to the phenomenon where output predictions for data outside the training batch can change during an update. This paper provided an in-depth study of the churn in training the policy and value networks and the interplay between these two dynamics. It proposed a simply regularization loss to suppress the churn and observed in experiments that the policy performance improve accordingly.

**Pros:**
- Most reviewers consider the paper provide a convincing motivation. The study on the churn concept in deep RL is relative new but important, and this submission is likely to inform future research along this line.
- The main contribution is the study on the interplay of the policy and value churn, and the solution to suppress it effectively.
- The proposed method is simple but effective and versatile to be applied to different Deep RL backbone methods. Either used for single network training or together as "CHAIN".
- The experiments not only show that the performance improves but also provide direct evidences that the proposed method indeed reduce churns.

**Cons:**
- Two reviewers (DNMp, c7UJ) were concerned on the challenge of choosing the two additional hyperparameters. The authors proposed a new method to automatically choose the coefficients during the rebuttal and provided new experiments to justify its effecacy, which partially addressed the concern.
- Multiple reviewers suggests improvements of the writing, which authors have promised to include in the revision.
- Reviewer oLah considers the "churn" phenomenon to be closely related to the interference of neural networks, and the novelty of studying the churn problem and the proposed regularization loss is not as high as it claims. The authors' response did not fully resolve that concern.

Despite some minor concerns that are not fully addressed, all reviewers vote to accept this submission. I would encourage the authors to take the reviewers' comments seriously, incorporate the recommended changes in writing, explain the new automatic coefficient tuning method in the main paper with the new experiment results, and include the discussion of the missing related work as well as its relationship with interference in a prominent place in the text.

Additional comments from Reviewer Qjzk on the IQL discussion that may help the authors to improve their final version:

*I started the IQL discussion because that part of the paper was written poorly / ambiguously. When the authors' IQL experiment details became clearer to me, it was immediately apparent that they weren't training IQL + CHAIN the way they were training the other actor-critic methods. They were training it in one specific way (only penalizing the policy churn using their additional loss term, L_PC), while showing improved performance over the baseline. Because IQL is a special case, in that it permits decoupled training of the actor and critic networks, it allows us to inspect the effects of each CHAIN loss term in isolation (which I asked to authors to try out). The most interesting finding of these experiments was the following:*

**IQL + PCR > IQL (baseline) > IQL (sequential)**

**IQL + PCR > IQL (baseline)**, *as shared in the paper, was slightly surprising to me because IQL doesn't really have a "chain effect", but rather only a one-way influence of the ever-changing Q network on the actor's training. But since Q learning itself exhibits churn, one could imagine that this churn would "leak" into the policy training dynamics too, so it is plausible that IQL + PCR would also be helpful, and so this result is not too surprising.*

**I proposed IQL (sequential) to also break that one-way influence.** *Waiting to train the actor, until after the Q network is fully trained and frozen, should completely cancel any influence of the Q churn on the actor training, right? Apparently, the authors find that this performs even worse than IQL (baseline). So apparently exposing the actor network to some Q churn --- like in IQL (baseline) --- is, strangely, helping! I don't think this finding necessarily contradicts anything discussed in the paper, but I believe it does open up some new questions about churn, and is surprising in a good and exciting way, which makes me want to see this paper published.*
Decision: Accept (poster)