Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary

The paper studies interference and forgetting in the context of reinforcement learning (RL).  On the example of the Iterative Value Estimation family of algorithms, the authors define interference as the increase of the true Q target prediction error after updating Q function parameters. Since the true Q target is usually unknown, the authors propose to use the difference of squared TD-errors between updates as a proxy for interference. The paper further defines forgetting as the difference between the current agent performance and the best performance across all previous updates. On CartPole and Acrobot environments, the authors show a positive correlation between the proposed measures of interference and forgetting. They further qualitatively demonstrate that updates of the last layer weights result in higher interference compared to updates of intermediate layers. Finally, the authors propose an algorithm based on meta-learning for learning representations that minimize interference resulting in more stable return plots on Acrobot. 

Strengths
- The topic brought in the paper is important: the distribution shift and moving targets in RL are indeed problematic and often lead to instabilities during training and forgetting of high-return policies.
- The paper is written clearly and generally easy to follow.

Weaknesses
- The proposed algorithm is designed to explicitly minimize interference. However, seeking for minimizing interference / forgetting on its own might prohibit exploration. For example, an agent that achieves the worst possible returns and is not learning at all will have zero interference / forgetting. (Perhaps it is more reasonable to seek monotonic policy improvement?)
- The finding that changes in parameters of the last layer result in higher interference seems unsurprising as, generally, changes in the final layer parameters affect the output of a neural network more than changes in intermediate layers.
- The proposed method for learning representations is based on meta-learning. It is unclear whether the learning curves on Acrobot are more stable due to claimed minimization of interference or due to using a more powerful optimization method. An ablation study will improve the quality of the paper.
- The authors should consider using harder environments to make the experimental results more convincing.

Positioning relative to the literature
- One of the main methodological contributions of the paper is using squared TD-error as a proxy for measuring interference. At the end of section 4, the authors mention related methods based on first-order approximation of interference. However, no comparison with the methods was provided.
- Similarly, the authors do not compare with other methods that address the forgetting in the RL context. For example, [2] uses simple weight averaging for minimizing forgetting and achieves learning curves similar to the ones in Figure 4.
- Perhaps the authors should reconsider the term “interference”. The cited paper [1] uses the term “interference” for an inner product between gradient estimates evaluated at different data points and seeks, in contrast, to maximize interference.

Recommendation

The reviewer leans towards rejecting the paper. The discussed problem is important, however, the findings of the paper do not seem generally surprising. The authors claim that “there is no established online measure of interference for RL” but the resulting measure of interference is simply a difference between squared TD-errors before and after an update. The proposed measure of forgetting, being a difference of current returns and previous best returns, has limited novelty too. Moreover, the paper lacks a comparison with related work. Addressing the outlined weaknesses might increase the assigned score.

**

Post-rebuttal update: the score is increased from 4 to 5. See the response to the authors' comments.

**

[1] Bengio, Emmanuel, Joelle Pineau, and Doina Precup. "Interference and Generalization in Temporal Difference Learning." arXiv preprint arXiv:2003.06350 (2020).

[2] Nikishin, Evgenii, Pavel Izmailov, Ben Athiwaratkun, Dmitrii Podoprikhin, Timur Garipov, Pavel Shvechikov, Dmitry Vetrov, and Andrew Gordon Wilson. "Improving stability in deep reinforcement learning with weight averaging." In Uncertainty in artificial intelligence workshop on uncertainty in Deep learning. 2018.

review_title: An important problem but the contributions have limited novelty, no comparison to the related work

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes new measures to quantify interference in reinforcement learning at different granularity and show the correlation between interference and forgetting, and suggests to use extra representation loss to reduce interference. 

In general, the definition of  interference at different granularity looks reasonable and the relation with forgetting matches intuition. However, there are no experiments to show any difference between those interference, such as update interference,  iteration interference, and interference across iterations.  So what's point to define interference with different granularity if only one is used in the end?  

The authors propose using extra representation loss to reduce interference, but the motivation is rather vague.  As the experiments show that internal layers contribute much less to interference, why it is better to adjust internal layers by representation loss than adjust the last layer which has much more influence on interference?  Is there more solid justification to connect representation loss and interference?  

The authors also introduces three different types of representation loss and OML obviously outperforms the others, but there is no enough analysis about why OML is supreme and then no guidance of how to choose a proper representation loss.

As I'm not an expert in reinforcement learning, I will leave the novelty and significance to other reviewers to decide. 

Some minor issues:
1. Equations should be numbered.
2. Could authors elaborate a bit on how to get the last equation on page 4? As the last term in the right side hasn't appeared before. 
3. The coefficients of forgetting and interference in the subfigures of Fig. 1 should be provided.
 

review_title: Probably interesting idea but not convincing enough

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper studies the the interference problem under the API and FPI setting. It designs new measure for interference, and shows that the interference measure is correlated with forgetting. With the help of the interference measure, the paper studies the importance of the final layer of the neural network and proposes a new algorithm to mitigate interference. 

Overall, the paper is well-written but I have some questions regarding the logic of the paper. First, it's not super clear to me what is the role of the forgetting measure. Since the validation of interference measure is evaluated by forgetting, I'm assuming forgetting is a more commonly used measure (please correct me if I'm wrong). 
- If this is the case, then it seems the importance of interference measure is to come up with the SRNN and OML variants. But I don't get why we cannot use the forgetting measure for regularization, maybe some clarifications needed here. If the interference measure is not necessary for OML and SRNN, I don't quite get the necessarily of the interference measure. 
- If this is not the case, then why forgetting is an interesting measure to compare with? 

In addition, I think it may be easier to follow if the unused definitions can be moved to Appendix, e.g., Pointwise Interference and Interference Across Iterations are only mentioned when they're defined, On the other hand, I think it'll be helpful to move some contents from Appendix to Sec 6.2, since it is the place that explicitly take advantage of the interference measure. 

Lastly, there two minor points: 
- Line 4. the definition of iid is missing
- Algorithm 1: it seems the the definition of b_k is missing
review_title: An interesting paper studying the interference under an API and FPI setting, but more clarification required.  

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper studies the reason for interference, aka catastrophic forgetting, when using parametric models for Reinforcement Learning. The authors draw the connection with previous methods and introduce some reasonable measure of interference. Then, they introduce a method to explicitly address the problem of interference, showing some good empirical results w.r.t. selected baselines.

I think that this paper studies an interesting problem, but its analysis is a bit superficial and not supported by rigorous theoretical analysis. The proposed measures of interference make sense, but it seems to me that they are not reliable measures, especially in the considered online learning setting. In fact, the TD-error may increase significantly across several iterations because of the agent exploring unvisited states, and not only because of interference. Previous works, e.g. Prioritized Experience Replay (Schaul et al, 2016), show that a higher TD-error is actually desirable to guide exploration, so I'm unsure how the motivation behind this work relates with the literature. Moreover, since catastrophic forgetting is mostly problematic in deep RL, I'd have liked a stronger focus on deep RL, where the author could have tested the benefit of their approach on several algorithms based on the TD error, e.g. DQN, DDPG, TD3, SAC. It is also confusing how the authors refer to FQI as an online algorithm. FQI is known to be a batch RL algorithm, i.e. an offline algorithm where a fixed dataset of transitions collected by another agent is available, even though it can be used for iterative policy updates, but this is not the typical scenario of FQI.

Without an extensive empirical analysis, and considering the absence of a rigorous theoretical analysis motivating the proposed interference measure, I think this paper is not ready for publication and I encourage the authors to improve it, especially showing stronger and more significant empirical evidence over representative baselines, perhaps even considering some multi-task and/or lifelong learning problems where interference constitutes a bigger issue.

Pros
------
* The considered problem is interesting for a broad group of researchers in RL;
* The presentation is clear enough.

Cons
-------
* The proposed measures are intuitive, but not supported by strong theoretical guarantees. In particular, the effect of exploration on the behavior of the TD error is not accurately discussed;
* Absence of deep RL experiments to show the effectiveness of the proposed approach on more challenging problems.


Post-rebuttal feedback
-------------------------------
I thank the authors for their reply. I still think that this paper has the major problem of presenting results about interference that are not strong enough to be published. In particular, I agree when the author say that "the paper tries to bring conceptual clarity to this important topic, and provide a clear empirical methodology to measure interference and understand correlations to forgetting", but still I think that the overall contribution consists "just" of one (of several other possible ones) intuitive method to measure interference, without a solid motivation. To me, this paper is promising but should present more significant results to have a stronger impact. I think the options are only two: stronger theoretical motivation, larger empirical analysis especially considering deep RL. Between the two, I consider the second option the best.
review_title: Interesting work, with weak theoretical motivation

Meta-review: The paper investigates interference in reinforcement learning and introduces a novel measure that can be used in value-based methods.
Although the reviewers acknowledge that the paper has merits (the topic is relevant and the paper is well written), they feel that the contribution is not sufficiently supported by either a theoretical or empirical analysis. The authors' responses have solved some of the reviewers' concerns, but they agree that this paper is not ready for publication in its current form.
I encourage the authors to update their paper following the reviewers' suggestions, in particular by improving the empirical analysis where comparisons with alternative methods (e.g., AVI/API methods that introduce regularization) need to be added.
Decision: Reject