The Phenomenon of Policy Churn
Authors: Tom Schaul
Andre Barreto
John Quan
Georg Ostrovski
Venue: NeurIPS 2022 Accept (2022)
Keywords: Reinforcement Learning
Exploration
Deep learning
Deep RL
Policy space
Stability

Abstract: We identify and study the phenomenon of policy churn, that is, the rapid change of the greedy policy in value-based reinforcement learning. Policy churn operates at a surprisingly rapid pace, changing the greedy action in a large fraction of states within a handful of learning updates (in a typical deep RL set-up such as DQN on Atari). We characterise the phenomenon empirically, verifying that it is not limited to specific algorithm or environment properties. A number of ablations help whittle down the plausible explanations on why churn occurs to just a handful, all related to deep learning. Finally, we hypothesise that policy churn is a beneficial but overlooked form of implicit exploration that casts $\epsilon$-greedy exploration in a fresh light, namely that $\epsilon$-noise plays a much smaller role than expected.

The Phenomenon of Policy Churn
Tom Schaul
André Barreto DeepMind London, UK
John Quan DeepMind London, UK
Georg Ostrovski DeepMind London, UK
DeepMind London, UK
{tom, andrebarreto, hatpannosssroossk///eemmm.aaddd
Abstract
We identify and study the phenomenon of policy churn, that is, the rapid change of the greedy policy in value-based reinforcement learning. Policy churn operates at surprisingly rapid pace, changing the greedy action large fraction of states within a handful of learning updates (in typical deep RL set-up such as DQN on Atari). We characterise the phenomenon empirically, verifying that limited to specific algorithm or environment properties. A number of ablations help whittle down the plausible explanations on why churn occurs, the most likely one being deep learning with high-variance updates. Finally, we hypothesise that policy churn is potentially beneficial but overlooked form of implicit exploration, which casts e-greedy exploration in fresh light, namely that e-noise plays much smaller role than expected.
not
The Phenomenon
Reinforcement learning (RI.) involves agents that incrementally update their policy. This process is driven by the objective of maximising reward, and based on experience that the agent generates via exploration. The sequence of policies To..... ........ AT usually starts from a randomly initialised policy To and aims to end at a near-optimal policy тт Re т². Ideally, steps in that sequence (TR TT++1) are policy improvements that increase expected reward.
This paper studies the amount of policy change that goes along with such policy update process (for a definition, see Section 1.1). In particular, it makes the core observation that policy change in pructice (as illustrated in some typical deep RI. settings) is orders of magnitude larger than could have been expected, and stands in contrast to various reference algorithms (Sections 1.2 and 3.3).
Key observation 1: The greedy policy changes much more rapidly than you probably think."
"As coarse magnitude for the impatient reader: in typical run of DQN on Atari, the greedy policy changes 10% of all states after single gradient update (Figure and Section 1.2).
We dub this phenomenon "policy churn" to highlight that most of this policy change may be unnecessary. We study the phenomenon in depth, determining the range of deep RL scenarios appears in. fleshing out its properties, and in the process narrowing the space of potential causes and mechanisms involved using set of ablations (Section
Our second key message relates the phenomenon of churn to exploration, specifically in the context of e-greedy exploration (Section 2), with some speculative ramifications in Section 4.
Key observation 2: Policy churn a significant driver exploration."
"This holds both in the that redueing chur reduce performance, and in the sense that explicitly adding noise hecomes unnecessary the presence churn (i.c., e = is viable).
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Asterix
Boxing
DDON early
Breakout
Freeway
Gopher
Gravitar
H.E.R.
Pac
R202 early
Pong
Orbert
Riverraid
Seaquest
Sking
Space Invaders
Average policy change 17 per gradient update
Figure 1: Average amount of policy change W (Eq. 2) update, in two deep RL agents (DoubleDON and R2D2). Points are averages over seeds, on one of (colour-coded) Atari games. "Farly" denotes the first 25% of training, "late" denotes the final 25%; observe that churn magnitudes drop late in training, but only for R2D2. See Figures 15 and 16 in the appendix for more me-ggrineed results.
1.1 Defining policy change
A policy is function from states E S to distribution over actions E A, where for the purposes of this paper A is discrete. We quantify the local, per-state policy change between policies π and x' using the moved probability mass (i.e., the total variation distance):
W(T, x = 2 Σ ((aas) (c|ss)
(1)
which satisfics 0 < W(T, s'(s) S 1. When π and n are greedy policies derived from state- sate-action- if the arg max action in
value function- that is, (s)) arg max, ((s, a)tnhen W [||s) W(n, state 8 changes upon replacing by the function q' underlying n., and WiT, >l|s) = otherwise. Similar reasoning applies more generally when both and x are deterministic policies. We can aggregate policy change across all states, weighted by state-distribution d:
W (T,, == Rasa [(T,, n' rl|s)],
(2)
where reasonable choice for d the empirical state distribution encountered during training (which non-stationary, but it could also be the stationary distribution of a fixed policy, the uniform distribution across states, as discussed some of the toy scenarios below). For greedy policies (and uniform d), W W(T, x') simply the fraction of states where an arg max switch occurred.
For settings where policy performance stabilises at some point = of training (e.g.. hitting a performance plateau, converging optimal behaviour), two additional metrics may be of interest, namely the cumulative policy change W Wo. until that point', and the average policy change after that point W (which could zero, e.g., if the process converges):
P-1
W U.P = (Tg,
W
lim sup
W P.T
(3)
T T - P-P
1.2 Quantifying the phenomenon
Given an initial and final policy, the process with the minimum amount of policy change is an oracle that jumps from To to in a single step (P : 1). By construction, this can incur a policy change of at most one unit, W P S 1. In value-based deep RI. agents, a natural definition for the the induced greedy policies (s)s)) arg max, 20, (s, a) where 0z
sequence of policies the parameters of the Q-function at iteration k. In agents that use a target network (inducing л) that is an older copy of the online network (inducing n'), it easy to measure W (T, T') by comparing their
Note that the granularity of updates (e.g.. batch size) determines how many intermediate policies are considered, which in turn affects the measured magnitude of policy change in learning processes that have churn.

---

Asterix
Boxking
Breakout
Frecway
Gopher
154
20003
302
30
320
5003
52-
120
-50
200
Pong
2.0
Q*bert
20k
1:0
-12
Riverraid
Seaqueest
Skinng
Space invaders
Venture
20k
20c
-15g
tpoodh ndum
154
150
135
5tlp c>
1oth c50 chum
-20
o:ly churn
25%
retter
-300
203
Nillon
NEllon framme
18llion
Figure Impact of churn on exploration. Plots show performance of DoubleDQN on Atari, with four variants of exploration: green is the unmodified baseline (e = 0.01), gold changes = 0, blue has reduced churn by acting with the target network, and red shows the effect of both of these changes together (no more exploration, and a corresponding performance collapse in most games). Thin lines depict individual seeds (3 per setting), thick lines are averages, smoothed over 10M frames.
arg max actions at the points in training where the target network lags behind by just one update. Figure 1 shows typical values for W on a few Atari games, estimated by comparing the policies induced by online and (one update old) target networks, on batches of experience sampled from the agent's replay buffer. It is worth emphasising that with such rates of average policy change, of W s 10% per update, the magnitude of whole-lifetime change becomes enormous: across training, an agent like DQN, which performs 10? updates, changes its greedy action million times in each state (on average).
second striking result is the amount of policy change in late training, when the performance of the policy longer changes (in the case of PONG it is arguably optimal): there is still a change of Wt x 5% per update (Figures 1 and 14). This highlights that a lot (maybe most) of policy churn is not directed at policy improvement; revisit this in Section 4.2.
Is this unexpected? We conducted an informal survey of over 50 deep RL practitioners, including three of the inventors of DQN [27], for their estimate on how rapidly the policy changes in a typical Q- learning based setup. The question was: For the greedy policy to change in 10% of all states, how many learning updates does it take? (or equivalent). The median response was 1 (000 updates, with answers varying between and 000 updates. This deviates by three orders of magnitude from the empirical value of 1 update (or W z 0.1, see Figure 1).
Policy change in other settings. To appreciate the large empirical magnitudes of (cumulative) policy change we observe in deep RL, example, classic dynamic programming techniques such as value iteration or policy iteration [44], when applied to toy RL domains such as FourRooms, CATCH, or DeepSea [33], accumulate W Wo.P& Rs 1, not much more than the single-step oracle (see Appendix B.4). The two main differences from deep RI. are their tabular nature (no function approximation, FA) and non-incremental updates. is possible to construct tabular settings with much larger policy change, with either incremental updates (Appendix A.3) or bootstrapping (Appendix A.4). A minimalist example with non-linear FA and incremental updates is supervised learning the MNIST dataset (the "policy" in this case are the predicted label probabilities). Training a digit classifier to convergence accumulates Wo.P ~ 10, that is, the average input goes through label switches (see Appendix B.5). None of these examples comparisons; so in Section 3.3 we construct
contrast them with few alternative settings. For
are fully satisfying, as they are not spectrum of algorithmic variants that spans from tabular policy iteration (without churn), via tabular Q-learning an approximation of DQN (with realistic magnitudes of churn).

---

2 The Exploration Effect
We now turn to the impact of policy churn on agent behaviour: What happens when acting according to policies produced by a learning process that induces rapidly changing policies? In other words, what is the effect of policy churn on exploration? While each individual greedy policy would lead to very narrow set of experience in (nearly) deterministic environment like Atari, the fact that the greedy policy changes so rapidly (in 10% of states per update, with one update every 16 environment frames in DoubleDQN) makes for a broad data distribution. And in many cireumstances this is sufficient for good performance, even in the absence of any other form of exploration, such as stochasticity introduced via an e-greedy policy. Figure 2 shows this across range of Atari games: compare green (baseline) and gold (c = 0) curves."
Conversely, removing (some of the) churn in the behaviour during training, which can be done by acting with the target network (updated only every 120 frames in the DoubleDQN agent) instead of the online network, sometimes reduces performance even in the presence of e-greedy exploration (blue in Figure 2).² Additionally, we show that performance often collapses completely when both forms of exploration are removed (e = 0 and no churn, in red). Figure 2 compares all four variants of exploration, indicating that the two sources of exploration have different contributions in different games.
Sufficient exploration with = O. The perhaps unintuitive observation of successful exploration with purely greedy policies has been made before, albeit implicitly. In particular, in the presence of certain alternative exploration methods such as noisy nets [15], no significant additional advantage is obtained from using e > 0 [20]. Other works containing experimental variants with = 0 demonstrated successful training in this setting [35, 39]. but did not highlight the result.
Consistent behaviour and Thompson sampling. In considering the potential exploration benefits of a rapidly changing policy. it is worth qualitatively contrasting the resulting behaviour with that of an e-greedy policy with e > O. The latter generates high-frequency dithering, with uncorrelated random action decisions at consecutive states and the effect of most exploratory actions likely undone by the following greedy action [11]. By contrast, policy-churn -olic--ehur-indueedd exploration can be expected to generate temporally correlated, consistent exploration (necessary, though perhaps not sufficient, to perform "deep" exploration, as in ensemble and Thompson sampling methods [45, 32]). On the other hand, while c-greedy exploration is explicitly unbiased in action space, policy churn likely prefers exploration across near-optimal actions (with respect to the current value function). This may be beneficial in some settings, for example when some actions are deadly, and high prevents long episodes. It may also be detrimental in others, where e >> 0 helps the agent get unstuck.
3 Potential Causes
With the presence and impact of the churn phenomenon established, this section aims to provide additional depth. First, we look at the generality of the effect in Section 3.1. Second, we conduct investigations into the sensitivity of the phenomenon, with Section 3.2 showing ablations to the large-scale deep RL agents (more material in Appendix A), and Section 3.3 taking the complementary approach of interpolating between dynamic programming and a DQN approximation on toy domain. Finally, Section 3.4 synthesises the findings and postulates some compatible underlying mechanisms.
3.1 Breadth of prevalence and non-causes
To judge the importance of the phenomenon, we need to establish whether it is specific to a narrow range of settings, or prevalent in a variety of domains, algorithmic variants, and haper-parameetrs. It turns out that this is easy to do, because the effect is very much not a subtle one. In fact, policy churn is present in two very different deep RL agents, namely DoubleDQN [47] and (a variant
In line with prior work, all our DoubleDQN experiments preserve the decaying e-schedule in the first 2% of training. it is only zero after that initial phase; R2D2 experiments have such schedule.
"Acting with the target network also introduces latency on how fast newly learned knowledge can be exploited. To see how specifically this latency should have a negligible effect on performance, imagine shifting the r-axis of the blue curve by 120 000 frames the left, which would be an oracle ofteerr---ooo--fftlett the-th--fee future" variant.

---

Agent
DoubleDON
R2D2
Input
84 x 84 grayscale
210 x 160 RGB
Action set
minimal per game:
| S 18
full: |A| 18
Reward
clipped
unclipped
Neural net
feed-forward, 1.7M parameters
recurrent, 5.5M parameters dueling 5-step double Q-learning
Q-value head
regular
Update
1-step double Q-learning
Optimiser
RMSProp without momentum
Adam with momentum = 0.9 x 2560 prioritised (exponent 0.9), 192
Batch size
32 uniform, 8
Replay, replay ratio
Parallel actors
Mean W per update
9%
6%
Table The two agent setups considered differ in a number of properties. Given both settings result in similar churn, none of them seem critical. See Appendix B.1 and B.2 for details.
of) R2D2 [21], both widely used for training on Atari (see Appendix B. and B.2 for agent and environment specifications). Despite the large differences between the algorithms summarised in Table 1, the magnitude of policy change is surprisingly similar, indicating that it is unlikely that policy churn strongly depends on any of these specific choices.
The effect is also not specific to any environment: we measure similar magnitudes of policy change range of Atari games that vary in many dimensions, such action space, reward scale and sparsity, deadliness, etc. Furthermore, is present in all stages of training (see Figures and 15). Unsurprisingly it is highest in early learning, but it remains high during training and even after evaluation performance converges or stabilises.
3.2 Ablations
Redundant actions. A simple factor that could explain policy churn are redundant actions, i.e., when nominally different actions have the same outcome (in all, or large fraction of state space). This property varies widely by environment (see also [31]), but we observe similar levels of policy change across them (Figures 1 and 15). Also, when exposing the full Atari action set in all games, creating explicit global redundancy, the churn magnitudes are not affected much (Figure 16). Appendix A.2 looks at the fine-grat fine-grained aspect of which actions tend to be swapped for which others, and finds no structure easily related to the (known) equivalence relations (Figure 10). In other words, most arg changes not happening between equivalent actions.
Small action gaps. Another potential factor for large policy churn with greedy policies can be the interplay between FA and small action gaps (difference between largest and second-largest action values): small approximation error suffice for sub-optimal actions to overtake optimal ones. This hypothesis predicts that value learning methods inducing larger action gaps (e.g., Advantage Learning (AL, [3]) which artificially lowers the values of sub-optimal actions) could reduce policy churn. Figure (right) shows that indeed policy churn decreased substantially by AL, correlating with an increase of action gaps (which consistently grows under AL, Figure 17). Curiously, this does not seem to severely diminish the remaining policy churn't 's effectiveness for exploration: Figure 18 shows successful training of an AL-DQN with 0.
Non-stationary state distribution. Another explanatory hypothesis for the observed large magni- tude of policy churn relates it to the non-stationary data distribution caused by the policy generating the data, which is evolving. Even when the agent converges to near-optimal performance, like in PONG, may happen that the policy keeps changing on states where actions are inconsequential (where multiple actions have near-equal value), causing non-stationarity in the data distribution and thereby driving further policy churn. To test this, we utilize the "forked tandem" setting from [34], in which gegh-performing policy trained on the stationary data distribution generated by its initial
GGienn the high churn in (preliminary) experiments on DM-Lab [1], as shown in Figure 25, we also consider it unlikely that the phenomenon is specific to the Atari setting.
5

---

Advantooo cap (bow. top actions)
Baseline
Advanlage learning
10
Double Q-learning
Stationary inputs
Stationary inputs and targets
All but top layers frozen
10
9
All but top layers frozen All but top layer frozen (linear)
10-3
10
10"1 Average policy change 97
Average policy change
Figure 3: Left: Ablations in the "forked" DoubleDQN setting (5 games, 3 seeds each). Training progresses as normal until 50M frames, after which either the data distribution is fixed ("stationary") or part of the neural network is frozen; "stationary targets" denotes regression onto Monte-Carte Carlo returns. The only large effect on policy change is when switching from deep learning to linear FA (red). For more detail see Figures 19, 20, and 22. Right: Relation between policy change and advantage gaps. Different points correspond to different games, seeds and stages of learning. This shows that the action-gu catton-gap-inan -nccressngs algorithm (purple) reduces policy change; and also how, given an algorithm, advantage gaps correlate with churn. For more detail see Figures 17 and 18.
snapshot (see below for more details). As can be seen in Figures 3 (left) and 19, the high level of policy churn is still preserved in this stationary-data regime, ruling out data non-stationarity as the main driver of the phenomenon.
Non-stationary targets. Temporal difference (TD) learning can give rise to another form of non- stationarity, the bootstrap targets change at the pace of the target value function, aspect that is preserved even in the forked tandem setting. To sidestep this, a simple control experiment uses the same setup but with Monte-Carlo returns learning targets, turning it into pure policy evaluation via supervised regression (with noisy targets). The results (Figures 3, left, and 20) show a similar level of churn to the Q-learning updates, indicating that the phenomenon is not specific to TD-based algorithms.
Decoupled acting and target networks. To assess how much policy churn is beneficial for explo- ration, we devise the following setup: A greedy policy based on the agent's 's target network drives behaviour, and we vary the frequency with which is synchronized the online network. To avoid conflation with an adverse effect on learning stability, we keep the regular target network update frequency constant, while using an additional acting network, periodically copied from the online network, for behaviour generation alone. Figure 21 shows that acting network updated more than every (000 gradient updates achieves most of the exploration benefits of acting with the online network, though in some games higher frequencies yield further benefit. This implics that the amount of policy change needed for exploration is much smaller than what is generated by DQN's learning process (otherwise the observed collapse would happen at much smaller update intervals).
Self-correction and the tandem effect. In [34] a phenomenon dubbed the "tandem effect" was observed: the failure of deep RL agent adequately learn from the training data generated by different instance of the same agent, highlighting the importance of self-correction by interactively generated data from the policy being trained. One of their settings, the "forked tandem", starts with copy of a high-performing policy and uses data sampled from its stationary distribution to continue training; even this apparently benign scenario leads instability and potential collapse of the trained policy. Policy churn may provide partial mechanistic explanation for the origin of the instability, showing that rapid policy change can be expected at all stages of training. The observation that the trained policy changes on a significant proportion of states at every update (performed on negligibly small sample, a single minibatch of 32 state transitions) supports the hypothesis from [34] that erroneous extrapolation or over vee-generalizztion may play key role in causing deviation and instability and producing the tandem effect the absence of corrective training signal from self-generated data. Analogously to results in [34] we observe that the magnitude of policy churn is highly correlated with the depth of the trained function approximator, further supporting this hypothesis; see Figures (left) and 22, which show specifically how policy change drops dramatically in linear FA regime.

---

Value keration
Tabular Q g-leamiin
Q-lea ming with layer MIP
Regression on q² with layer MLP
Q-learning with layer MILP
DON-like with RMSProp
DON-like with SGD
DQN like with Adam
10
10
10
10
10'
10
10
10
10a
10'
10
Cumulative policy change until convergence Wi.P
Aver age policy change after convergence W
Convergence stop
Figure 4: Ablations of what aspects drive policy change in CATCH. Unless specified each FA variant uses the SGD optimiser. Here "DQN-like" is "Q-learning with 3 layer MLP" with replay and mini-batches. Error bars denote the interquartile range over 100 seeds.
3.3 Detailed case study: CATCH
CATCH is toy environment where the total number of states is small enough to be amenable to ground-truth dynamic programming approaches using the explicit matrix of transition probabilities, while providing an observation space that requires non-linear FA to represent the optimal value function. We construct a spectrum of settings that all learn the optimal policy after some /' iterations.
For each measure cumulative policy change Wo.P and average change after convergence W* (Eq. 3). Main results are shown in Figure 4. Sitting at one end of the spectrum, exact value iteration converges after = 10 steps with W o:r = 0.09 (because the initial policy is random, and the optimal policy has ties in most states). The next simplest setting tabular Q-learning with incremental updates (here and elsewhere, hyperparameters like the learning rate are tuned for fast convergence to optimal performance, see Appendix B.3 for details). The next steps add in non-linear FA (shallow with 1 hidden layers, then deep, with 3). At the other end of the spectrum an approximation to DQN which includes experience replay, mini-batches, deep neural network, and an advanced optimiser (RMSProp [46]). Figure 4 also shows intermediate variations such as supervised regression to q", and different optimisers (e.g., Adam [23]); additional results Appendix A. Overall, the experiments in this section show that, among the factors we considered, the presence of function approximation is by far the aspect that correlates the most with the occurrence of the phenomenon of policy churn.
3.4 Mechanistic hypothesis
In an attempt to synthesise all the evidence presented so far, we propose that the observed policy churn primarily a result of components that need to be present jointly:
1. Non-linear, global function approximation (such as deep neural networks), where each update can affect all states and all actions.
2. A learning process with high amount of noise. This could have multiple sources: stochastic optimisation (c.g. small batch sizes, large learning rates), noisy learning targets, non-stationary data (or targets, as with bootstrapping). an imbalanced data distribution (e.g., seeing some actions more often than others).
In other words, the rapid policy change dubbed churn is the symptom of high variance updates to global function approximator, and neither aspect in isolation is enough promote it.
'A compounding effect could be mismatch between the regression loss in value space that drives the learning process, and policy space, which is what matters for performance and for exploration (and measured here). While supervised learning does not have this mismatch, it could conceivably have large "policy change" too. An in de-eeph treatment exceeds the scope for this paper, but preliminary results on MNIST (Appendix B.5) indicate no large churn in the supervised setting.
7

---

Where do we go from here?
4.1 Learning at the edge of chaos
Deep learning has well-known trade-off between speed of learning and stability that incentivises tuning the learning dynamics to be near the edge of chaos [10]. The presence of policy churn could enrich this picture in two ways for the case of deep RL. First, to the extent that rapid policy change helps drive exploration (Section 2), this is an additional incentive to keep learning dynamics sufficiently noisy. Second, to the extent that value- based learning relies on self-correction [34], that is, the actions to be corrected need to be picked by a greedy policy, this also encourages a large amount of policy change. Circumstantial evidence for these is that value-based RL tends to require additional stabilisation mechanisms (e.g., target networks), less common in policy-based RI..
4.2 Policy null-spaces
Policy change is necessary for policy improvement, but does not necessarily imply change in performance. We define the space of policies that have the same value as a reference policy æ (in all states) as null space Null(T). One way to quantify it is the diameter
[Nul(TT]
max
W(T,x""
o' .a (NNu((()
as the largest policy change possible between any pair of policies within it. Typical scenarios with large null spaces have states in which actions have no effect (e.g., move actions while falling), multiple actions with the same effect (globally, or locally in part of state space), or multiple paths leading to the same outcome (e.g., up-then-leff vs. left-then-up). These scenarios are common among the environments typically studied in deep RL. Appreciating that policy null spaces can be large makes the policy churn phenomenon more palatable, helping us reconcile that agents changing their mind million times about the best action (on average in each state) can have excellent performance. particularly interesting null space is the one around the optimal policy Null(r"): )): when it is large, there are many "safe" ways to keep changing the policy (possibly by lot) after converging to maximal performance, which is what we observe on, e.g., PONG (see Figure 14), at least when >> 0.
helping
4.3 Off-policy corrections in the presence of churn
Most types of off- off-policy correction are based on the gap between the data- generating behaviour policy J4, and a target policy т. When doing multi-step step updates, conservative methods truncate trajectories to bootstrap early compared to on- on-policy experience [25]. To obtain the benefit of multi-step value propagation, the average truncation length cannot be too short, and thus silent assumption is that and π do not differ too frequently. Empirically however, multi-step back-ups without any off-policy correction can be surprisingly effective [20]. We can now re-interpret these findings through the lens of policy churn: as the greedy policy changes much faster than expected, even a slight latency (a few updates) between the parameters of the data-generating policy and the current target policy leads to massive truncation effect. If consecutive greedy policies are (approximately) within the null-space of each other, the benefits of an uncorrected multi-step update may outweigh its cost. It also suggests the possibility of new off- policy algorithms that exploit the knowledge of the churn phenomenon, by truncating in less aggressive fashion, motivated by the intuition that majority of rapidly changing policies lie within an (approximate) null space of each other (see Appendix A.5).
4.4 The social dynamics of research
If policy churn is indeed a hidden form of exploration, how did come about? It seems unlikely that a useful mechanism emerges completely by chance. An intriguing possibility is that policy churn the effect of a gradual process of natural selection [12]. The hypothesis is that, if policy churn provides benefits, algorithms that display some level of it would be favoured over their counterparts in the inevitable engineering work surrounding the design of large-scale agents. this view, RI. practitioners play the role of "nature" exerting selective pressure that shapes algorithms across time. This process could be completely unconscious to the researchers involved: agents under-performing
As epitomised in the heuristic to tune the learning rate the largest value that does not explode.
8

---

due to weak exploration would be discarded in favour of agents that explored better using some form of (hidden) policy churn. Over time, the multiple degrees of freedom of large-scale agents thyper-parameters, network architecture, etc.) would be tuned to reflect, just the right amount of churn: enough to help in exploration, but not too much to make the overall learning process unstable.
As appealing as the above hypothesis may be, as of now we do not have any evidence to support it. Still, it is worth considering, as it raises many interesting questions. Are there other hidden effects that have been selected for over the years? How many good design choices got discarded because they happened not to promote policy churn (or other similarly hidden effects)? Is the Al research community narrowly focused on handful of design templates that happen to induce some ill-understood dynamics? This sort of question did not arise in the past, when agents were simple enough that we could keep track of their functioning at the finest level of detail. Now that deep RL agents have reached a certain level of complexity, any design choice may have cascade effect whose consequences we do not anticipate. This creates the perfect environment for the sort of selective process described above. Acknowledging this possibility and being aware of it may be an important step toward unveiling hidden effects, and perhaps turning them into more purposeful design.
5 Related work
phenomenon in the literature that is related to policy churn, but not equivalent to it, is that of policy oscillation or chattering [16, 4, 48]. Bertsekas and Tsitsiklis [4] define the greedy region of function q E Q as the set of functions in Q that induce the same greedy policy as q. Each greedy region has an associated "fixed point": the value function of the greedy policy induced by the functions in it. Conversely, every value function is the fixed point of a greedy region. It is well known that the only value function that belongs to its own greedy region is the optimal value function q" [4]. However, when function approximation is used, projecting value functions onto the FA space may create cycles that repeat indefinitely, a phenomenon known policy oscillation. We do not think policy oscillation is a likely explanation for policy churn, because the approximators used in our experiments should have the capacity approximate any value function to reasonable level of accuracy. This is corroborated by Figures 3 and 4 and Appendix A. which show neural networks with more or wider layers exhibit more churn. We believe it is more instructive update of the approximator as moving the value function across the boundaries of greedy regions. As discussed, in many cases this has no effect on the agent's s performance, since the policies associated with neighbouring greedy regions may belong to each other's null space (Section 4.2).
think of each
Another body of related work is the literature stochastic gradient descent, and how a relatively high amount of stochasticity can be beneficial to optimisation by overcoming local optima and converging to flatter optima with better generalisation properties [22, 37, 24]; this is such a prominent effect that in some cases it is even beneficial to inject additional noise with Langevin dynamics [50]. More loosely related are studies of learning in animals, exhibiting large drift in synaptic or representation space [30, 38, 26, 13, 42], as well as evidence for highly variable behaviour policies that can get consolidated through salient dopamine events [9].
6 Conclusions and Future Work
Revisiting interpretations. Nine years after the introduction of DQN [27], there are still phenomena in talue-based deep RL that remain to be understood, with this paper putting the spotlight on one of them that lies at the intersection of learning and exploration dynamics. In particular, we hope that an awareness of the churn phenomenon will make researchers revisit some good ideas that may have been prematurely disregarded or under-valued, either because their promised exploration effect was too entangled with learning dynamics and the resulting churn, or because they improved the stability of learning dynamics at the expense of reduced exploration that undid the overall gains.
Churn beyond value-based RL. An obvious follow-up question is whether policy churn is also an important effect beyond value-based algorithms. We could imagine that actor-critic algorithms incur much less policy change, because stochastic policies change more smoothly, or because various penalty terms keep the updated policy from deviating too much from its precursor. If that were the case, would indicate that exploration in these agents differs as well, possibly with complementary advantages and vantages Similarly, various instances of model-based RL [29, 43] may have

---

less policy change, because the planning process could mitigate some of the function approximation effects. We leave these investigations future work, but include one preliminary result (Figure 26) that seems to hint at churn being high in some actor-critic agents as well.
Explicit and controllable churn. If policy churn is indeed valuable and non-trivial exploration mechanism, then may be costly deliberately abandon it, especially its effect turns out to be complementary to simpler noise-based exploration mechanisms. Ideally we would want an explicit and controllable mechanism that produces the same kind of consistent, non-harmful exploration behaviour, but without such tight entanglement with the learning dynamics. The core benefit of such a division of labour would that practitioners could study, change or tune the learning and exploration processes separately, without having to make arbitrary trade-offs (e.g., around stability, diversity, representations) that inevitably arise when they are considered in combination.
Acknowledgments
The ideas presented here were refined in discussion with numerous of our DeepMind colleagues. Will Dabney, Joseph Modayil and Matteo Hessel helped improve the paper with detailed feedback. and we thank David Silver, Diana Borsa, Miruna Pislar, Claudia Clopath. Vlad Mnih, Jurii Kemaev, Junhyuk Oh, Bilal Piot, Greg Farquhar, Dan Calian, Hado van Hasselt and Alex Pritzel for their input. We also thank the anonymous NeurIPS reviewers for their suggestions, as well as the many RLDM and ICML attendees whom we put our guesstimate survey question to.
References
[1] C. Beattic, J. Z. Leibo, D. Teplyashin, T. Ward, M. Wainwright, H. Küttler, A. Lefrancq. S. Green, V. Valdés, Sadik, et Deepmind lab. arXiv preprint arXiv:1612.03801 2016.
[2] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
[3] M. G. Bellemare, G. Ostrovski, A. Guez, P. Thomas, and R. Munos. Increasing the action gap: New operators for reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, 10, Feb. 2016.
[4] D. P. Bertsekas and J. N. Tsitsiklis. Neuro-Dynanie Programming. Athena Scientific, 1996.
[5] J. Bradbury, R. Frostig. P. Hawkins, M. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke,
J. VanderPlas, S. Wannerman-Milne and Q Zhang. JAX: composable transformations of Python+NumPP programs, 2018.
[6] D. Budden, M. Hessel, Kemaev, Spencer, and F, Viola. Chex: Testing made fun, in JAX!,
2020.
[7] D. Budden, M. Hessel, J. Quan, S. Kapturowski, K. Baumli, S. Bhupatiraju, A. Guy, and M. King. RLax: Reinforcement Learning in JAX, 2020.
[8] A. Cassirer, G. Maron, T. Sottiaux, M. Kroiss, and E. Brevdo. Reverb: An efficient data storage and transport system for MI. research, 2020.
[9] C. Clopath, L. Ziegler, E. Vasilaki, L. Büsing, and W. Gerstner. Tag- crnogggrrronssliiaaiinn a model of early and late font-termnootenntiaton and depression. PL.S computational biology. 4 4(12)xe1000248, 2008.
[10] J. M. Cohen, S. Kaur, Y. Li, J. Z. Kolter, and A. Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. arXiv preprint arXiv:2103.0006. 2021.
[11] W. Dabney, G. Ostrovski, and A. Barreto. Temporally-extended e-greedy exploration. In 9th
International Conference Learning Representations (ICLR'21) 2021.
[12] C. Darwin. The origin of species by means of natural selection. John Murray, 1859.
[13) D. Deitch, Rubin, and Y. Ziv. Representational drift in the mouse visual cortex, bioRxiv,
2020.
10

---

[14] L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, et al. IMPALA: Scalable distributed Deep-RL with importance weighted actor- learner architectures. arXiv preprint arXiv: / /80200061, 2018.
[15] M. Fortunato, M. G. Azar, B. Piot, J. Menick, I. Osband, A. Graves, V. Mnih, R. Munos, D. Hassabis, O. Pietquin, C. Blundell, and S. Legg. Noisy networks for exploration. arXiv preprint arXiv: a
2017.
[16] G. J. Gordon. Stable function approximation in dynamic programming. In Machine Learning Proceedings 1995, pages 261-268. Elsevier, 1995.
[17] T. Hennigan, T. Cai, Norman, and I. Babuschkin. Haiku: Sonnet for JAX, 2020.
[18] M. Hessel, D. Budden, F. Viola, M. Rosca, E. Sezener, and T. Hennigan. Optax: Composable gradient transformation and optimisation, in JAX!, 2020.
[19] M. Hessel, M. Kroiss, A. Clark, I. Kemaev, Quan, T. Keck, F. Viola, and H. van Hasselt. Podracer architectures for scalable reinforcement learning, arXiv preprint arXiv:2104.062: 2021.
[20] M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-secondl AAAI conference on artificial intelligence, 2018.
[21] S. Kapturowski, G. Ostrovski, W. Dabney, J. Quan, and R. Munos. Recurrent experience replay in distributed reinforcement learning. In International Conference on Learning Representations, 2019.
[22] N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang. On large-batch train- ing for deep learning: Generalization gap and sharp minima. arXiv preprint 16X:::1609044836 2016.
[23] D. P. Kingma and Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 6980, 2014.
[24] B. Kleinberg, Y. Li, and Y. Yuan. An alternative view: When does sgd escape local minima? In International Conference on Machine Learning, pages 2698 2707. PMLR, 2018.
[25] T. Kozuno, Y. Tang. M. Rowland, R. Munos, S. Kapturowski, W. Dabney, M. Valko, and D. Abel. Revisiting peng's q (A) for modern reinforcement learning. In International Conference on Machine Learning, pages 5794-5804. PMLR, 2021.
[26] D. Marks and M. Goard. Stimulus-deeeddent representational drift in primary visual cortex. bioRviv, 2020.
[27] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing Atari with deep reinforcement learning. arXiv preprint arXiv: 2013.
[28] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533, 2015.
[29] T. M. Moerland, J. Broekens, and C. M. Jonker. Model-based reinforcement learning: A survey. arXiv preprint arXiv:2006. 167 12, 2020.
[30] G. Mongillo, S. Rumpel, and Y. Loewenstein. Intrinsic volatility of synaptic connections challenge to the synaptic trace theory of memory. Current Opinion in Neurobiology, 46:7-13, 2017. Computational Neuroscience.
[31] M. J. Nelson. Estimates for the branching factors of Atari games, arXiv:2 00.303885..... 2021.
arXiv preprint
[32] I. Osband, C. Blundell, A. Pritzel, and B. Van Roy. Deep exploration via bootstrapped DQN. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
[33] I. Oshand, Y. Doron, M. Hessel, J. Aslanides, F. Sezener, A. Saraiva, K. McKinney, T. Lattimoree C. Szepesvari, S. Singh. et al. Behaviour suite for reinforcement learning. arXiv preprint 2019.
Ar0iv:1908.00
[34] G. Ostrovski, Castro, and W. Dabney. The difficulty of passive learning in deep reinforce- ment learning, arXiv preprint
2021.

---

[35] M. Pislar, D. Szepesvari, G. Ostrovski, D. Borsa, and T. Schaul. When should agents explore?
In International Conference on Learning Representations (ICLR), 2022.
[36] Quan and G. Ostrovski. DQN Zoo: Reference implementations of DQN-based agents, 2020.
[37] S. Ruder. An overview of gradient descent optimization algorithms. arXiv preprint 1609.04460 2016.
[38] M. E. Rule and T. O'Leary. Self-healing neural codes. bioRxiv, 2021.
[39] T. Schaul, D. Borsa, D. Ding, D. Szepesvari, G. Ostrovski, W. Dabney, and S. Osindero. Adapting behaviour for learning progress, arXiv preprint arXiv: 2019.
[40] Schaul, G. Ostrovski, Kemaev, and D. Borsa. Return-based scaling: Yet another normalisa- tion trick for deep RL. arXiv preprint arXiv:2105.05344 2021.
[41] T. Schaul, J. Quan, I. Antonoglou, and D. Silver. Prioritized experience replay. In International Conference on Learning Representations (ICLR), 2016.
[42] C. E. Schoonover, S. N. Ohashi, R. Axel, and A. P. Fink. Representational drift in primary olfactory cortex. Nature, 594(7864)5441-5466 2021.
[43] Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lock- hart, D. Hassabis, T. Graepel, et al. Mastering Atari, Go, Chess and Shogi by planning with learned model. Nature, 588(7839):604 609, 2020.
[44] R. Sutton and A. Barto. Reinforcement learning: An introduction. MIT press, 2018.
[45] W. R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):285-294, 1933.
[46] Tieleman, G. Hinton, et al. RMSProp: Divide the gradient by running average of its recent
magnitude. COURSERA: Neural networks for machine learning, 4(2):26-31, 2012.
[47] H. Van Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double Q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
[48] P. Wagner. Policy oscillation is overshooting. Neural Networks, 52:43 61, 2014.
[49] Z. Wang, T. Schaul, M. Hessel, H. Hasselt, M. Lanctot, and N. Freitas. Dueling network architectures for deep reinforcement learning. In Proceedings of The 33rd International Conference Machine Learning, volume 48, 2016.
[50] M. Welling and Y. W. Teh. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-II), pages 681 1688. Citeseer, 201 1.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Section 3 discusses the generality
and applicability, Section the ramifications.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Section 4.4 is specifically about the impact on the social dynamics of research.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as a URL)? [No] Our implemen- tation relies on some proprietary code.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Appendix B.
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes] Many our experiments have seeds across 15 games which is enough to draw meaningful conclusions. We report either error bars, show box-plots, or show all seeds individually.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix B.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [Yes]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [Yes] Nobody was paid.
13


Reviews:
[{"review_id": "wU792HrNQDA", "reviewer": "Reviewer_6hS6", "summary": "This work identified and studied the phenomenon of policy churn, which refers to the rapid change of the greedy policy in value-based reinforcement learning.\n\nThe authors first conducted experiments on several Atari games with two methods (DoubleDQN and R2D2), and the results in Figure 1 showed that each one update would cause about $10$% greedy policy changes of all states during training, while $5$% greedy policy changes when the performance of the policy no longer changes.\n\nThe authors then showed that the policy churn has some exploration effect. As shown in Figure 2, acting with the policies produced by a learning processing with high churn would make additional eps-greedy exploration kind of not necessary. And removing the churn would reduce performance.\n\nSection 3 used ablations to eliminate several potential reasons for policy churn to happen, and made some hypotheses that policy churn is a result of non linear function approximation and stochastic updates.", "questions": "1. It is claimed that \"removing the churn would reduce performance\". However, as shown in Figure 2, it seems that for many games the blue curves are comparable to green and gold ones? How is this concluded from this figure?\n\n2. The way the authors did to \"removing the churn\" is \"acting with the target network\". As noted this also changed the learning dynamics somehow rather than just \"removing the churn\". This seems to introduce a compounding effect: by \"acting with the target network\", the samples being collected are different, and thus it changes the policies after performing updates using the collected samples.\n\nIn a word, \"acting with the target network\" introduced other changes rather than \"removing the churn\", and it seems \"shifting the x-axis of the blue curve by 120000 frames to the left\" is not convincing to me to correct it. Therefore, this part of argument for \"policy churn has exploration effect\" is not convincing to me.\n\nPlease correct me if I misunderstood something or please clarify this matter.\n\n------update------\nThank you for the response. I would increase my score since I agree with the authors and other reviewers that discovering the policy churn phenomenon would be valuable and could inspire future study.", "limitations": "The experiments are conducted on publicly available benchmarks which have been widely used for years. I did not see negative societal impact of this work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. Policy churn of value-based RL methods is an unnoticed phenomenon, and it is to me an interesting and relevant result. \n\n2. The experiments and ablations are substantive, which I appreciate.\n\n3. The presentation is clear and easy to follow.\n\nWeaknesses:\n\n1. It remains still unclear as noted how the policy churn phenomenon can be useful for designing better algorithms.\n\n2. It is still largely unclear where the concrete root cause of the policy churn is.\n\n3. The argument that policy churn is a good way of exploration is not very convincing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "qIMUXYiuQ4n", "reviewer": "Reviewer_ZHm5", "summary": "This paper firstly identifies the policy churn phenomenon, which is not noticed by RL practitioners. The paper demonstrates that this phenomenon is related to deep learning. Furthermore, they found that this phenomenon is a benefit for exploration. ", "questions": "\nQuestions:\n1. In Fig. 2, what is a policy change for only churn? How does it compare the policy change of \"both $\\epsilon$ and churn\" and \"only $\\epsilon>0$\"\n\n2. Policy churn is not always a good thing. For example, there should be less policy churn when converging. Could the author discuss more on how to control policy churn when the policy converges? Could that improve the final performance if the policy churn is controlled?\n\n3. In line 323,  I don't know why the paper states that policy churn is complementary to simpler noise-based exploration. At least in Fig 2, combining noise-based and churn didn't perform best on all tasks (either does policy churn).\n\n4. in my view, policy churn is still different from noise-based exploration, as noise-based exploration imposes the largest probability of the most promising action. Could the policy churn attain these effects, i.e., controllable exploration? Did the author do experiments on that?\n\n\n\nMinor Problems:\n\n$\\bar{W}^+$ is a variable related to $P$. Would that be better to use a subscript with P?\n", "limitations": " ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n\nGenerally, I like the work of this paper. Policy churn is not noticed by RL practitioners previously. This paper did good work on this interesting phenomenon. It provided abundant experimental analysis.\nThe results make sense to me. It's expected in a dynamic programming setting that policy churn is not significant. In contrast, the noise in updating can result in this phenomenon. \n\nThe main effect that this paper study is to relate policy churn with exploration, which is interesting. However, there are some flaws in the statements (see questions below). Furthermore, I expect the author to discuss more whether policy churn takes effect on other elements of RL, e.g., policy improvement, and the quality of resulted policy.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "T2p5RbKByPc", "reviewer": "Reviewer_Q1cd", "summary": "This paper identifies and studies a phenomena they name policy churn; the fact that during the learning process in value-based RL algorithms, the greedy policy changes at a very rapid rate (in a single update the greedy policy changes in 10% of all states). They observe and justify empirically that this phenomena is a key driver of exploration in Atari games. They identify this phenomena as happening both in DoubleDQN and R2D2, which are fairly different value-based algorithms, demonstrating that the phenomena is fairly general. The investigate several possible causes of the phenomena, and while they don't fully explain it, they propose a possible hypothesis for the primary causes of policy churn (non-linear global function approximation combined with a highly noisy learning process). They provide an extended discussion on the implications and future directions of research from this phenomena.", "questions": "I don't have any low-level questions. My main high-level questions are mostly posed in the strengths and weaknesses section, in terms of asking for additional experiments.", "limitations": "While there is some implicit discussion of limitations throughout the paper, I think it would be beneficial to explicitly state that it's not yet clear that this phenomena is more general than off-policy value-based methods on Atari.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall I like the paper. I think exploratory and explanatory work like this is often undervalued, but is crucial to improving our understanding of how RL algorithms work. The paper is well-written and easy to follow, and clearly demonstrates an interesting phenomena, while backing it up with a wide variety of empirical experiments and investigations. The research claims made are sound, and appropriately cautious where there is not compelling evidence. For all these reasons, I'm in favour of accepting the paper.\n\nThe main weakness lies in the fact that the phenomena isn't entirely explained. I think this is too-high a bar to require for the paper's acceptance, but it would be great to have a more complete explanation of the phenomena.\n\nA second weakness that I think is possibly more easily addressable the lack of generality in the environments on which the phenomena is investigated. While Atari is a staple of the RL literature, I'm cautious of making general claims about RL learning from performing experiments on a single set of environments, which while diverse in some ways are very non-diverse in others. While I'd expect this same phenomena to hold for value-based methods applied in other environments, I'd prefer to have the prediction validated, and it seems like a relatively easy set of experiments to perform.\n\nI'd also be interested in seeing investigation of policy churn in on-policy methods (such as PPO), or in the policies of actor-critic methods for continuous control such soft actor critic or TD3. While I think the contribution is still valuable even if it's limited to off-policy value-based methods similar to DQN, showing that the phenomena is more general than that would be beneficial.\n\nI'm currently recommending accept, and would raise my rating to a strong accept if experiments were performed on other environments, either with similar or different methods, regardless of the outcomes of those experiments. I'd also raise my score if the mechanistic hypothesis in 3.4 was operationalised technically, and then investigated, although I'm not expecting that as it requires more work that can reasonably be expected in a rebuttal period.\n\nEDIT: I thank the authors for their response, and the addition of new experiments demonstrating the broad appearance of the policy churn phenomena. I raise my score to an 8 in light of this.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept