Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work identified and studied the phenomenon of policy churn, which refers to the rapid change of the greedy policy in value-based reinforcement learning.

The authors first conducted experiments on several Atari games with two methods (DoubleDQN and R2D2), and the results in Figure 1 showed that each one update would cause about $10$% greedy policy changes of all states during training, while $5$% greedy policy changes when the performance of the policy no longer changes.

The authors then showed that the policy churn has some exploration effect. As shown in Figure 2, acting with the policies produced by a learning processing with high churn would make additional eps-greedy exploration kind of not necessary. And removing the churn would reduce performance.

Section 3 used ablations to eliminate several potential reasons for policy churn to happen, and made some hypotheses that policy churn is a result of non linear function approximation and stochastic updates.
strengths_and_weaknesses: Strengths:

1. Policy churn of value-based RL methods is an unnoticed phenomenon, and it is to me an interesting and relevant result. 

2. The experiments and ablations are substantive, which I appreciate.

3. The presentation is clear and easy to follow.

Weaknesses:

1. It remains still unclear as noted how the policy churn phenomenon can be useful for designing better algorithms.

2. It is still largely unclear where the concrete root cause of the policy churn is.

3. The argument that policy churn is a good way of exploration is not very convincing.
questions: 1. It is claimed that "removing the churn would reduce performance". However, as shown in Figure 2, it seems that for many games the blue curves are comparable to green and gold ones? How is this concluded from this figure?

2. The way the authors did to "removing the churn" is "acting with the target network". As noted this also changed the learning dynamics somehow rather than just "removing the churn". This seems to introduce a compounding effect: by "acting with the target network", the samples being collected are different, and thus it changes the policies after performing updates using the collected samples.

In a word, "acting with the target network" introduced other changes rather than "removing the churn", and it seems "shifting the x-axis of the blue curve by 120000 frames to the left" is not convincing to me to correct it. Therefore, this part of argument for "policy churn has exploration effect" is not convincing to me.

Please correct me if I misunderstood something or please clarify this matter.

------update------
Thank you for the response. I would increase my score since I agree with the authors and other reviewers that discovering the policy churn phenomenon would be valuable and could inspire future study.
limitations: The experiments are conducted on publicly available benchmarks which have been widely used for years. I did not see negative societal impact of this work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper firstly identifies the policy churn phenomenon, which is not noticed by RL practitioners. The paper demonstrates that this phenomenon is related to deep learning. Furthermore, they found that this phenomenon is a benefit for exploration. 
strengths_and_weaknesses: 

Generally, I like the work of this paper. Policy churn is not noticed by RL practitioners previously. This paper did good work on this interesting phenomenon. It provided abundant experimental analysis.
The results make sense to me. It's expected in a dynamic programming setting that policy churn is not significant. In contrast, the noise in updating can result in this phenomenon. 

The main effect that this paper study is to relate policy churn with exploration, which is interesting. However, there are some flaws in the statements (see questions below). Furthermore, I expect the author to discuss more whether policy churn takes effect on other elements of RL, e.g., policy improvement, and the quality of resulted policy.

questions: 
Questions:
1. In Fig. 2, what is a policy change for only churn? How does it compare the policy change of "both $\epsilon$ and churn" and "only $\epsilon>0$"

2. Policy churn is not always a good thing. For example, there should be less policy churn when converging. Could the author discuss more on how to control policy churn when the policy converges? Could that improve the final performance if the policy churn is controlled?

3. In line 323,  I don't know why the paper states that policy churn is complementary to simpler noise-based exploration. At least in Fig 2, combining noise-based and churn didn't perform best on all tasks (either does policy churn).

4. in my view, policy churn is still different from noise-based exploration, as noise-based exploration imposes the largest probability of the most promising action. Could the policy churn attain these effects, i.e., controllable exploration? Did the author do experiments on that?



Minor Problems:

$\bar{W}^+$ is a variable related to $P$. Would that be better to use a subscript with P?

limitations:
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper identifies and studies a phenomena they name policy churn; the fact that during the learning process in value-based RL algorithms, the greedy policy changes at a very rapid rate (in a single update the greedy policy changes in 10% of all states). They observe and justify empirically that this phenomena is a key driver of exploration in Atari games. They identify this phenomena as happening both in DoubleDQN and R2D2, which are fairly different value-based algorithms, demonstrating that the phenomena is fairly general. The investigate several possible causes of the phenomena, and while they don't fully explain it, they propose a possible hypothesis for the primary causes of policy churn (non-linear global function approximation combined with a highly noisy learning process). They provide an extended discussion on the implications and future directions of research from this phenomena.
strengths_and_weaknesses: Overall I like the paper. I think exploratory and explanatory work like this is often undervalued, but is crucial to improving our understanding of how RL algorithms work. The paper is well-written and easy to follow, and clearly demonstrates an interesting phenomena, while backing it up with a wide variety of empirical experiments and investigations. The research claims made are sound, and appropriately cautious where there is not compelling evidence. For all these reasons, I'm in favour of accepting the paper.

The main weakness lies in the fact that the phenomena isn't entirely explained. I think this is too-high a bar to require for the paper's acceptance, but it would be great to have a more complete explanation of the phenomena.

A second weakness that I think is possibly more easily addressable the lack of generality in the environments on which the phenomena is investigated. While Atari is a staple of the RL literature, I'm cautious of making general claims about RL learning from performing experiments on a single set of environments, which while diverse in some ways are very non-diverse in others. While I'd expect this same phenomena to hold for value-based methods applied in other environments, I'd prefer to have the prediction validated, and it seems like a relatively easy set of experiments to perform.

I'd also be interested in seeing investigation of policy churn in on-policy methods (such as PPO), or in the policies of actor-critic methods for continuous control such soft actor critic or TD3. While I think the contribution is still valuable even if it's limited to off-policy value-based methods similar to DQN, showing that the phenomena is more general than that would be beneficial.

I'm currently recommending accept, and would raise my rating to a strong accept if experiments were performed on other environments, either with similar or different methods, regardless of the outcomes of those experiments. I'd also raise my score if the mechanistic hypothesis in 3.4 was operationalised technically, and then investigated, although I'm not expecting that as it requires more work that can reasonably be expected in a rebuttal period.

EDIT: I thank the authors for their response, and the addition of new experiments demonstrating the broad appearance of the policy churn phenomena. I raise my score to an 8 in light of this.

questions: I don't have any low-level questions. My main high-level questions are mostly posed in the strengths and weaknesses section, in terms of asking for additional experiments.
limitations: While there is some implicit discussion of limitations throughout the paper, I think it would be beneficial to explicitly state that it's not yet clear that this phenomena is more general than off-policy value-based methods on Atari.
review_title: null

Meta-review: metareview: The paper identifies a hitherto unknown phenomenon in value-based reinforcement learning called policy churn. All the reviewers agree that this is a very interesting phenomenon, and that the paper studies the phenomenon comprehensively. All reviewers also agreed that this paper is likely to inspire many follow-up works. Understanding the causes and harnessing policy churn can potentially significantly impact the deep RL state-of-the-art. 
confidence: Certain
Decision: Accept