Measuring and mitigating interference in reinforcement learning
Authors: Vincent Liu
Adam M White
Hengshuai Yao
Martha White
Venue:  (2021)
Keywords: Reinforcement Learning
Representation Learning

Abstract: Catastrophic interference is common in many network-based learning systems, and many proposals exist for mitigating it. But, before we overcome interference we must understand it better. In this work, we first provide a definition and novel measure of interference for value-based control methods such as Fitted Q Iteration and DQN. We systematically evaluate our measure of interference, showing that it correlates with forgetting, across a variety of network architectures. Our new interference measure allows us to ask novel scientific questions about commonly used deep learning architectures and develop new learning algorithms. In particular we show that updates on the last layer result in significantly higher interference than updates internal to the network. Lastly, we introduce a novel online-aware representation learning algorithm to minimize interference, and we empirically demonstrate that it improves stability and has lower interference.

Under review as a conference paper at ICLR 2021
MEASURING AND MITIGATING INTERFERENCE IN RE- IEFORTEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Catastrophic interference is common in many network-based learning systems, and many proposals exist for mitigating it. But, before we overcome interference we must understand it better. In this work, we first provide definition and novel measure of interference for value-based control methods such as Fitted Q Iteration and DQN. We systematically evaluate our measure of interference, showing that it correlates with forgetting, across variety of network architectures. Our new interference measure allows to ask novel scientific questions about commonly used deep learning architectures and develop new learning algorithms. In particu- lar we show that updates on the last layer result in significantly higher interference introduce novel online-aware
than updates internal to the network. Lastly, representation learning algorithm to minimize interference, and we empirically demonstrate that it improves stability and has lower interference.
INTRODUCTION
Generalization a key property of reinforcement learning (RI.) algorithms with function approx- imation. An agent must correctly generalize its recent experience to both states it has not yet en- countered and other states it encountered in the past. Generalization has been extensively studied in supervised learning, inputs are sampled iid from fixed input distribution and the targets are sampled from fixed conditional distribution.
The distribution of training data is often not iid. When learning from stream of temporally corre- lated data, as in RI., the learner might fit the learned function to recent data and potentially overwrite previous learning- for example, the estimated values. This phenomenon is commonly called inter- ference or forgetting in RI. (Bengio et al.) 2020; Goodrich] 2015 Liu et al.) 2019 Kirkpatrick et al.) 2017 Riemer et al. 2018). The conventional wisdom is that interference particularly problematic in RL, even single-task RL, because (a) when agent explores, it processes sequence of obser- vations, which are likely to be temporally correlated; (b) the agent continually changes its policy, changing the distribution of samples over time; and (c) most algorithms use bootstrap targets (as in temporal difference learning), making the update targets non-stationary.
It is difficult verify this conventional wisdom, as there is no established online measure of in- terference for RI. There has been significant progress quantifying interference in supervised learn- ing (Chaudhry al., 20188] Fort et al. 2019; Kemker et al. 2018 Riemer al.) 22018), with some empirical work even correlating interference and properties of task sequences (Nguyen et al. 2019). and investigations into (un) forgettttee examples in classification (Tneva al. (019). In RL, recent efforts have focused on generalization and transfer, rather than characterizing or measuring interfer- ence. Learning on new environments often results in drops performance on previously learned environments (Farebrother et al.) 2018 Packer et al. 2018, RRaeewarrnn et al.] 2017] Cobbe et al.) 2018). DQN-based agents can hit performance plateaus in Atari, presumably due to interference. In fact, if the learning process is segmented in the right way, the interference can be more precisely characterized with TD errors across different game contexts (Fedus et al.] 2020). Unfortunately this analysis cannot be done online learning progresses. Finally, recent work investigated several different possible measures interference, but did not land on a clear measure (Bengio et al.) 2020).
In this paper advocate for simpler approach to charactering interference in RL. In most sys- tems the value estimates and actions change on every time-step conflating many different sources of non-stationarity, stochasticity, and error. If an update to the value function interferes, the result of

---

Under review as a conference paper at ICLR 2021
that updated might not manifest in the policy's performance for several time steps, if at all. Inter- ference classically refers to an update negatively impacting the agent's 's previous learning cearning-eroding the agent's 's knowledge stored in the value function. Therefore it makes sense first characterize interference in the value function updates, instead of the policy or return. We define interference in terms of prediction error for two common approximate dynamic programming algorithms, ap- proximate policy iteration and fitted Q iteration. Most value-based deep RL algorithms are based on these two algorithms. The interference is defined as the change in prediction errors, which similar to previous definitions of interference in supervised learning. Additionally, our approach yields an online estimate of interference, which can even be directly optimized.
In this work, we provide a clear justification for the use of differences in squared TD errors as the definition of interference. We highlight the definitions of interference at different granularities, and the utility of considering different statistics to summarize interference within iterations versus over time. We evaluate our interference measure by computing the correlation to a forgetting metric, which reflects instability in control performance. We show that high interference correlates with forgetting, and simultaneously show interference and forgetting properties across a variety of archi- tectures and optimization choices. We then use measure to highlight that updates to internal layers of the network- the representation contribute much less to interference than updates on the last layer. This motivates the design of a new algorithm that learns representation online that plicitly minimize interference. We conclude with a demonstration that this algorithm does indeed significantly improve stability and reduce interference.
PROBLEM FORMULATION AND LEARNING ALGORITTMS
In reinforcement learning (RL), an agent interacts with its environment, receiving observations and selecting actions to maximize a reward signal. We assume the environment can be formalized as a Markov decision process (MDP), An MDP is a tuple (S, A. Pr, R, y, dg) where S is a set of states, A is an set of actions, Pr: S x A x S + [0. 1] is the transition probability, R : x A is the reward function, 7 E [0, 1J a discount factor, and do is the initial distribution. The goal of the agent is to find a policy π S
+ I
+ [0, 1] to maximize the expected discounted sum of rewards.
Given a fixed policy म, the action-value function Q S x A R is defined Q*(s,a) E|ER- 2 R = s, Ar = a). where Ro+1 = R(S, Ar, S4+1), S4+1 ~ (t(-)S;,A)) and actions are taken according to policy tion can be obtained using the Bellman operator for action values T RlS]iiAl + R|SIxIAI; (TFQ)(s, a) := Eves Pr(s' s, a) RRs. a, 8') + E.EE T<alls)) ((sssss((sssssssss a')]. Q is the unique solution of the Bellman equation T=Q=Q.
A ~ ((.).)). Given a policy the value func-
The optimal value function Q" is defined as (s) == sup, Q(8 a). with Tt the policy that is greedy w.r.t. Q'. Similarly, the optimal value function can be obtained using the Bellman optimality operator for action values T R || + |AS] (TQ)(s,a) Eves Pr(s'|s,a) a) [R(s, a, s') + maxa'EA (((s'aa'), Q" the unique solution of the Bellman equation TQ = Q. We can use neural networks to learn approximation Qo to the optimal action-value, with parameters O.
In this work, restrict our attention to Ieerttie Value Estimation algorithms. These are algorithms where there is an explicit evaluation phase with a fixed policy, where the agent has several steps Teval improve its value estimates. Two examples of such algorithms are Approximate Policy Iteration (API) and Fitted Q-Iteration (FQI) (Ernst et al.) 2005). In API, for the current policy Tk. the agent updates its estimate of TT by taking Toval steps in the environment and performing mini-batch update from replay buffer on each step using the Sarsa update:
0.11 02 + ad. Vo, Qo, (S., Ar)
where d, = -Pe1 + -Q0, (St+1, (8(+ Qa, At).
In FQI, the policy and targets Qi are held fixed for Teval steps, with these fixed targets used as a regression target in the update. Again, mini-batch update update from a replay buffer is used on each step as above, but with a different 8 = Re+1 + maxx Qu(S a') - Qo. (S., A.). The procedure for both algorithms is summarized in Algorithm
2

---

Under review as a conference paper at ICLR 2021
Algorithm Iterative Value Estimation: A General Framework for API and FQI
Initialize weights 0g. Initialize empty buffer of size B. for ( + 0, 1, 2.... do If l mod Teval 0 then Qx Qa update ma to be greedy w.r.t Qs, by to be e-greedy Choose ag ~ by (st). observe ($t+1, re+1). and add the transition to the buffer Sample mini-batch of transitions B, from the buffer and update the weights:
0.+1 f 02 + a [B,| [(sa,,a)) 6(0;; 8,0,s,00 s, a, ,, (s, a)
= + Qo (8', (x())))) - Qo (s, and for FQI: 8(0; s, s, = r + max Qx(s', aa) Qa. (s.
where for API: 8,,0,,,ar,,
DEFINING INTERFERENCE IN VALUE ESTIMATION
In this section, we define interference for Iterative Value Estimation algorithms. Because these methods have evaluation phase which corresponds one iteration, we can more clearly define interference within one iteration. We discuss interference at four different levels of granularity.
Within each iteration-in each evaluation phase-w value estimates improve or degrade? The evaluation phase more similar to a standard prediction problem, where the goal is simply improve the estimates of the action-values towards clear target. Let f be the target function, which either (*),,a) = QT* v, for API or I (*(s,a) E|R + max,' Qs( a s, al for FQI.
ask: did the agent's knowledge about its
Pointwise Interference At the most fine-grained, we can ask if an update, going from 0g to 0 resulted in interference for specific point (s, a). The change in accuracy at s, a after an update
Accuracy Change((s, a), Or, 0.+1) == (s, - ( a)) - (f*(s, Qe, (s, a))²
where if this number is negative it reflects that accuracy improved. This change resulted in interfer- ence if positive, and zero interference if it negative, and so we have
Pointwise Interference((s, a), Or, 02+1) == max (Accuracy Change((s, a), O., 0.+1) .0)
Update Interference At a less fine fine-graince level, we can ask if the update generally improved our accuracy-our knowledge in our value estimates-acce across points.
Update Interference(O,, 0++1) max (E(s, [Accuracy Change((S, A), 0++1))+++1)))) .))
where (s, a) are sampled according to distribution d, such as from a buffer collected experience.
Notice that this differs from the expected Pointwise Interference. There settings where they could produce notably different values. For example, an agent could have high positive and negative Accuracy Change that cancel. The Update Interference reflects that, average, the agent's 's knowl- edge has not changed: it improved in some places, and degraded in others. The expected Pointwise Interference, on the other hand, would be high, because for some points interference was high. We focus first on Update Interference, since it is the coarser measure; future work is to look in a more fine-grained way at Pointwise Interference.
Both Pointwise Interference and Update Interference are about one step. At an even higher level, can then ask how much interference we have across multiple steps, both within an iteration and across multiple iterations. At this higher level, it becomes more sensible to consider upper percentiles, to ask if there was significant interference within an iteration and across iterations. For this we take expectations over only the top percentage of values. In finance, this is typically called the expected tail loss or conditional value at risk. Previous work RL (Chan et al. 2020) has used conditional value at risk to measure the long-term risk of RL algorithms.
Iteration Interference reflects if there was significant interference in updating during the evaluation phase (an iteration). Even few update steps having significant interference within iteration could cause significant instability; an average over the steps might wash out those few significant steps. We therefore define Iteration Interference for iteration using expectation over the top 10% of values
Iteration Interference(k) B[X]X X 2 Percentilen (X) (X)) for X Update Interference(Orr.e OT+1k)
3

---

Under review as a conference paper at ICLR 2021
where T is the time step in the iteration k uniformly distributed and Percentileu.9 is the 0.9- percentile of the distribution of X. Other percentiles could be considered, where smaller percentiles average over more values and percentile of 0.5 gives the median.
Interference Across Iterations reflects if an agent has many iterations with significant interfer- ence. Once again, even few iterations with significant interference could destabilize learning; expectations over tails are more suitable than over all iterations. For iteration K a random variable,
Interference Across Iterations [[X|X 2 Percentiley. (X)] for X Iteration Interference(K) (K)
These definitions are quite generic, assuming only that we have well-defined targets for the evalu- ation phase and an algorithm that proceeds in iterations. Though we have only discussed API and FQI, the algorithm DQN also fits well into this class-and and so could be analyzed-hheausee because the use of target networks mimics FQI. The primary difference is that the policy changes within iteration. This is not logistically an issue, as updates are off-policy from a replay buffer; but, it does add an additional changing variable. We therefore focus experiments on API and FQI.
4 APPROXIMATING ACCURACY CHANGE AND UPDATE INTERFERENCE
The primary difficulty now is estimating the Accuracy Change, which involves the true values Q" for API and expectations over next state and reward for FQI. With a simulator, these can fact be estimated. For small experiments, therefore, the exact Accuracy Change could be computed. More generally, the cost is prohibitive, and approximations are needed. In this section, we motivate the use of TD errors reasonable approximation.
We focus on finding a useful proxy measure for Update Interference, where the simply need our approximation be reflective of the average of Accuracy Change, rather than get it correct for particular point. Our goal is to find a proxy measure that is easy to compute and that at least reflects the same sign the Update Interference. For FQI, the target in Accuracy Change corresponds to the expected TD error, E 5 = 8, A = a). for the defined for FQI. For API, the expected TD error is the Bellman error: E[Ø|S A = s, A = a). The term Q= a) does not match the target f* a) = Q a). Fortunately, there is quite lot of theory showing that the Bellman error provides an upper bound on the value error (Williams) 1993), and further that using Bellman errors is sufficient to obtain performance bounds for API (Munos 2003 2007 Farahmand al., 2010).
8, A a] T T*Q((, a) Q(s, a), where T T*Q(s,
E,, [R +
Therefore, unified choice to approximate Accuracy Change for both API and FQI is to approximate the expected TD errors. can get an unbiased sample of these TD errors, but the square of these TD errors does not correspond to the squared expected TD error (Bellman error). Instead, there is residual term, that reflects the variance the targets (Antos et al. 2008)
E = A a] [5IS = A a]² + Var, R QolS', ,'){S = A a).
When we consider the difference in TD errors, after an update, for (s, a), we get
=5 s, (E((0,) -K] 8, s, A= -E.10(041]]
E, (600))
a)²
+ Varx 1R++Q.. (o.
= Var [R (S', A') a=
For given (s, we would not expect the variance of the target to change significantly. When subtracting the squared TD errors, therefore, we expect these residual variance terms to cancel. When further averaged across (s, a), it is even more likely for this term to be negligible. If the environment deterministic, then this variance is already zero and there no approximation.
The use of TD errors for interference is related to previous interference measures based on grudient alignment. To see why, notice if we perform an update using one transition (st, (t, Ft,, s(), then the interference of that update to (s, a, 6') is 6² (0(0411. s, a, rr," s') 8² (²)O,, series expansion, we get the following first-order approximation assuming a small step-size ac
s'). Using Taylor
00 (0, a, 8,, 8'T (0.41 0.) a (0,5s,,,, s,,), 0 (0,; 8t, a, F1,
This approximation corresponds to gradiens alignment, which has been used to learn neural net- works that more robust to interference (Lopez-Paz al. 2017 Riemer al. 2018). They
4

---

Under review as a conference paper at ICLR 2021
measure if this dot product is greater than zero, determine if there is transfer between two sam- ples; they generally encourage these dot-products to be positive. Other work used gradient cosine similarity, to measure the level of transferability between tasks (Du et al. 2018). and to measure the level of interference between objectives (Schaul et al., 2019). A somewhat similar measure was used to measure generalization in reinforcement learning (Achiam al. 2019), using the dot product of the gradients of Q functions VoQe, (3t, at) VeQu, (s, a). This measure neglects the direction of the gradients, and measures both positive generalization as well as interference.
Gradient alignment has a few disadvantages, as compared to using differences in the squared TD errors. First, as described above, it is actually a first order approximation of the difference, introduc- ing further approximation. Second, it is actually more costly to measure, since it requires computing gradients and taking dot products. Computing Update Interference on buffer of data only requires one forward pass over each transition. Gradient alignment, on the other hand, needs one forward pass and one backward pass for each transition.
MEASURING CATASTROPHIC INTERFERENCE AND FORGETTING IN RL
In this section, we empirically show that our Interference Across Time measure is correlated with forgetting, where agent performance drops. We define forgetting at each iteration as the difference between the best performance achieved before this iteration, and the performance after the policy improvement step. Precisely, let (a.))_do II (s, a)] be the agent performance after policy im- provement step at iteration where dg is the start-state distribution, where a random action taken in the first step. We estimate this value using 100 rollouts. Forgetting from iteration l is defined as
Iteration Forgetting(k)) == max ((s.)) (1la)) (s.a)] ((s.a) [**++1 (s, a)].
i= i=l...,k
As before, we take the expected tail over all iterations, to measure if the agent suffered from signif- icant forgetting. If even few iterations involve forgetting, and most steps do not, we should still consider Forgetting to be high. We therefore define Forgetting across iterations as
Forgetting == X 2 (rrcentileo.((X)) for X = Iteration Forgetting( (
We chose two environments: CartPole and Acrobot. In CartPole, the agent tries keep a pole balanced, with a positive reward per step. We chose CartPole because RL agents have been shown exhibit forgetting
this environment (Goodrich] 2015). In Acrobot, the agent has to swing up from a resting position to reach the goal, receiving negative reward per step until termination. We chose Acrobot because the physical dynamics are similar to Cartpole, but induces different learning dynamics: instead of starting from good location, it has to explore to reach the goal.
We ran variety of agents to induce variety of different learning behaviors. We generate a set of API and FQI algorithms by varying buffer size E {1000, 5000, 10000]. hidden layer size E (64, 128, 256, 512) and number of steps in one iteration M (100, 200, 4003. Each algorithm has 400 iterations. Additionally, we include neural networks with tile coded inputs, which has been shown to break generalization in neural networks (Ghiassian et al. 2020). All experiments are averaged over 30 runs. A buffer for measuring Interference is obtained using reservoir sampling from a larger batch of data, to provide a reasonably diverse set of transitions.
We show the correlation plot between Interference and Forgetting in Figure I In all figures except Figure T (b), there is a strong correlation between our definitions of Interference and Forgetting. In Figure (b). the correlation is unclear for M=100, possibly because all algorithms do not perform well, with all suffering from similarly high Forgetting which reduces as M increases. We note a few clear outcomes. (1) Neural networks with a smaller hidden size have less interference and forgetting. (2) Large buffer sizes seem to help with small networks. (3) FQI has lower magnitude Interference and less Forgetting than API on both environments. (4) Tile coded inputs reduce Interference and Forgetting for API but not for FQI, possibly because API bootstraps in its targets unlike FQI.
6 MITIGATING INTERFERENCE VIA ONLINE-AWARD REPRESENTATIONS
is common to assume that the first layers of neural network play the role of producing a represen- tation, that transforms the inputs into form that (1) provides suitable abstraction and (2) facilitates
5

---

Under review as a conference paper at ICLR 2021
M--200
input_ type
Forgeti
obs
Forgeting
tc buffer_size
1000
R45 Interference
Interference
Interference
Interference
Interference
Interference
(a) API on CartPole
5000
(h) API on Acrobot M-200
Mo=100
Mo=200
M00000
10000
M-100
Mm400
hidden size
gaggeting
128
256
Interference
Interference
Interference
Interference
Interference
Interference
(c) FQI on CartPole
(d) FQI on Acrobot
Figure I: Correlation plot of interference and forgetting. Each point represents one algorithm aver- aged over 10 runs. Outliers (which have interference larger than and forgetting larger than 80 on CartPole) are excluded from the plots.
learning in the future (typically called transfer). In the online setting, it has also been noted that the representation plays a third role: improving learning under online updating (Sutton 1996. Liu jet al. 2019: Daved and White 2019). We view value function with a representation function and a linear weight Qw,a(s, a) == ( a) w, where w E IRd is the weights the last layer and Φ3 S x A - IRd is the representation learned by the network with weights B, composed of all the hidden layers in the network. The function Φ a) w corresponds the last layer in the network, with w the weights of the network.
two-part approximation
In this section, we first show that updates in the final layer result in significantly more interference than updates in internal layers. This matches recent insights that learning sparse representation, of- fline, can significantly improve stability in online learning (Liu et al. (011 Javed and | (2019). Beyond these works, it highlights that the effort spent mitigating interference should be on pro- ducing a representation that mitigates interference. We then design an online-aware representation learning approach, that directly minimizes interference.
6.1 COMPARING INTERFERENCE DUE INTERNAL LAYERS VERSUS THE FINAL LAYER
In this section, we study interference due to updates in the internal layers and the last layer. To study interference separately within the network, we use stochastic block coordinate de- scent to update and separately, shown in Algorithm 2 We first update B, and com- pute Update Interference((E441. we]. [8,, w.)) due to this update. Then we update w, and compute Update Interference( [8++1, We+1], [8e41, (e)) again. Finally, we compute the Update Interference( (Bt+te We+1], ) [S,, w.)) for the whole update, so that we can measure the percentage contribution from each of the separate updates.
We report the percentage contribution of interference in each layer in Figure 2 We found that the last layer exhibits more interference than the internal layers consistently across the size of neural networks. Note that for neural networks with hidden size 512 on CartPole. the internal layers contain about 264,000 parameters while the last layer only has about 2,000 parameters. However, the last layer still exhibits significantly higher interference than the internal layers. This suggests that the internal layers change slowly or they change the TD errors slowly during the optimization process. Examining the distributions of interference over acatinaction pairs (see Appendix B), we do in fact find that updates in the internal layers result in smaller magnitude interference.
Figure 2: Contribution to Update Interference due to updates of the internal lay- ers versus the last layer. The percentages are aver- aged over 10 runs.
env CartPole
env Acrobot
90%
Last layer Internal Layers
45%
0%
64
128
256
512
512
Hidden Size
Hidden Size

---

Under review as a conference paper at ICLR 2021
Algorithm 2 API with Online Representation Learning
Initialize weights 3, randomly. Initialize empty buffer B of size N for ( 4 1, ...N do
If t mod Toval = then Qx Qe update TTy by to Qx
Choose ag ~ by (st). observe ($t+1. re+1). and add the transition to the buffer Sample mini-batch of transitions B, from the buffer
Update 81+1, using B₁ and the representation learning loss (see Appendix A.4 W++1 We at [B&]] E(sanna)) ( We; 8, a, 8') V') Qi++i. (8,a)
6.2 AN ONLINE- AWARE REPRESENTATION LEARNING ALGORITHM
In this section, we develop an algorithm to explicitly learn representations that mitigate interference. The idea is simple, and actually allows variety of offline representation learning approach to be incorporated into API and FQI. The first step is to separate the update to the representation, using block coordinate update,
we used in the last section to measure interference in different lay- ers. The second is to incorporate a representation learning loss, which can be update as would be offline simply by using replay step from the replay buffer. This allows to both incorporate two recently proposed offline objectives: Sparse Representation NN (SRNN) which uses a distribu- tional regularizer on the hidden nodes (Liu al.) 2019) and Online-Aware Meta-Leaming (OML) which uses meta-learning approach to adjust the representation to minimizes the loss after n on- line updates (Javed and White 2019). To the best of our knowledge, this is the first time these two approaches have been extended to the online setting, which the representation being learned concur- rently with the value estimates! [
We summarize this generic approach in Algorithm 2 with more specific details for each representation learning approach in Appendix A.4 and A.5
We can adapt the objective from Javed and White (2019) minimize Update Interference in RL. The goal is to learn a representation to minimize Update Interference from updates. We can sample a random evaluation buffer B, from the larger replay buffer, that implicitly specifies the dis- tribution d in the Update Interference. Then
get Wi+n the weights after experience replay up- dates, for a fixed representation 3r. We get the resulting Update Interference(1ee, Wi+n], [s,, w.] = E (,,,,, W=++))² - 6:( wz)) The goal is to optimize the current representation 3, so it produces the lowest interference after these n updates to w,. Because we do not update w, from 3t in the second term, only need to adjust 3. for the first term. Effectively,
are adjusting 3r so that it allows for a better final We+n. This is precisely how OMI. was defined, but for a supervised loss. In this sense, in its standard form, is already implicitly minimizing interference.
6.3 EMPIRICAL COMPARISON
-60
We compare FQI, API- ORI, API-SRNN and API- OML on three classic con- trol problems. API-ORL simply involves learning 3 with the two stage pro- cedure described in Sec- tion 5.1, without special representation learning ap- proaches. Figure| 3 sum- marizes the performance in Acrobot. The improvement for API-OML is made even more apparent when looking Forgetting- which reflects stability. The results for Mountain Car and Cartpole can be found in Appendix C All results are averaged over 30 independent runs and learning curves and bar plots show standard error.
Return
Forgetting
40
API-SANN
20
API-OML
FQI
-100
API-ORL
API-SRNN
API-OML
FQI
0
200
400
Number of Iterations
(a) Total Forgetting across iterations.
(b) Expected return vs iterations.
Figure 3: Acrobot results, averaged over 30 runs with standard errors.
A related approach, Experiexeeeeece Replay (MER) Riemer et al. 2018). directly optimizes gradient alignment. MER does not explicitly leam a representation. Instead, updates the entire network in the inner loop and outer loop. As result, it is significantly more computationally intensive. This necessitates the use of first-order approximation, and we were unable to get it perform consistently.

---

Under review as a conference paper at ICLR 2021
Figure Bb suggests actually removes much of the interesting structure, which is particularly relevant when investigat- ing interference. Looking closer, Figure 4a shows the return per run, revealing that API-ORL has considerable problems learning and maintaining stable performance. API-OMI in comparison is substantially more stable and reaches higher performance. Figure [4b] shows the iteration interfer- ence. Overall API-OML exhibits far less interference. but does exhibit high interference in two runs. These runs exactly correspond to the two runs in Figure 4a where API-OML is less stable in the beginning of learning. We do not know exactly why OML struggles in 2 of the 30 runs, but our measure of interference allows us to pinpoint precisely when it occurs. Finally, in Figure [ajwe we see that API-OML exhibits significantly less forgetting than all three baselines, dramatically improving over API-ORL.
API-OML method is better than the baselines, but averaging
اللl
المطااللل
بللاالك
Interference
-10
-00
24
2.
Iteration number
Iteration number
API-ORL
API-OML
API-ORL
API-OML
(a) Learning curves for API-ORL and API-OML, one (b) Iteration Interference for API-ORL and API-OML. run per subplot.
Figure 4: Learning curves and Iteration Interference for API-ORL and API-OML in Acrobot.
7 CONCLUSION
In this paper, we proposed a definition of interference for iterative value-based methods, API and FQI, and justified the use of squared TD error to approximate this interference. We showed this in- terference measure is correlated with control performance. We additionally provided some insights into interference in deep reinforcement learning algorithms, namely that FQI has lower interference than API and that APi-which uses bootstrapping rather than the fixed targets FQI-benefits more from strategies that reduce generalization inputs. We then used the measure show that the interference in updates, for standard neural networks, is due more to updates in the final layer than due updates internal layers. This motivated extending API and FQI to include an strategy to do online-aware representation learning, where the representation is explicitly trained to mitigate interference. The new algorithm provides simple mechanism convert two existing offline rep- resentation learning approaches to the online setting. We concluded with demonstration that the resulting algorithm, called API-OMI, significantly improved stability, and exhibited low interfer- ence under measure well as low forgetting.
These results highlight several promising avenues for improving stability in RL. One potentially surprising outcome was the instability, within a run, of standard method like API. The learning curve for API actually looked reasonable, and without examining individual runs, this instability would not be obvious. This motivates re-examining many of agents with alternative measures, like forgetting and other measures of stability. also highlights that there an exciting opportunity to si significantly improve our agents, by focusing these robustness improvements.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Joshua Achiam, Ethan Knight, and Picter Abbeel. Towards characterizing divergence in deep q- learning, arXiv:I903. 08894, 2019.
András Antos, Csaba Szepesvári, and Rémi Munos. Learning near-optimal policies with bellman- residual minimization based fitted policy iteration and single sample path. Machine Learning, 71(1):89-129, 2008.
Emmanuel Bengio, Joelle Pineau, and Doina Precup. Interference and generalization in temporal difference learning. In International Conference on Machine Learning. 2020.
Stephanie C.Y. Chan, Anoop Korattikara, Sam Fishman, John Canny, and Sergio Guadarrama. Mea- suring the reliability of reinforcement learning algorithms. In International Conference on Learn- ing Representations, 2020.
Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings the European Conference on Computer Vision, 2018.
Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying generaliza- 2018.
tion reinforcement learning. arXiv preprint
Yunshu Du, Wojciech M Czarnecki, Siddhant Jayakumar, Razvan Pascanu, and Balaji Lakshmi- narayanan. Adapting auxiliary losses using gradient similarity. arXiv preprint arXiv: ar212:18/2.02224, 2018.
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6(Apr):503-556, 2005.
Amir-massoud Farahmand, Csaba Szepesvári, and Rémi Munos. Error propagation for approximate policy and value iteration. In Advances in Neural Information Processing Systems, 2010.
Jesse Farebrother, Marlos C Machado, and Michael Bowling. Generalization and regularization in dqn. arXiv preprint arXiv:1810.00123, 2018.
William Fedus, Dibya Ghosh, John D Martin, Marc G Bellemare, Yoshua Bengio, and Hugo Larochelle. On catastrophic interference in atari 2600 games. arXiv preprint arXiv:2002.12499 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. In International Conference on Machine Learning, 2017.
Stanislav Fort, Pawel Krzysztof Nowak, and Srini Narayanan. Stiffness: A new perspective on generalization in neural networks. arXiv preprint arXiv:1901.0000 1901.09411, 2019.
Sina Ghiassian, Banafsheh Rafiee, Yat Long Lo, and Adam White. Improving performance re- inforcement learning by breaking generalization in neural networks. In International Conference on Autonomous Agents and Multiagent Systems, 2020.
Benjamin Frederick Goodrich. Neuron clustering for mitigating catastrophic forgetting in supervised and reinforcement learning. 2015.
Khurram Javed and Martha White. Meta-learning representations for continual learning. In Ad- vances in Neural Information Processing Systems, 2019.
Ronald Kemker, Mare McClure, Angelina Abitino, Tyler L Hayes, and Christopher Kanan. Mea- suring catastrophic forgetting in neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska Barwinska, al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 2017.

---

Under review as a conference paper at ICLR 2021
Vincent Liu, Raksha Kumaraswamy, Lei Le, and Martha White. The utility of sparse representa- tions for control in reinforcement learning. In Proceedings of the AAAI Conference on Artificial 2019.
Intelligence, volume 33, pages
David Lopez-Paz et al. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, 2017.
Rémi Munos. Error bounds for approximate policy iteration. In International Conference on Ma- chine Learning, 2003.
Rémi Munos.
Performance bounds in Lp-norm for approximate value iteration. SIAM journal on control and optimization, 2007.
Cuong V Nguyen, Alessandro Achille, Michael Lam, Tal Hassner, Vijay Mahadevan, and Ste- fano Soatto. Toward understanding catastrophic forgetting in continual learning. arXiv preprint arXiv:1908. 01091, 2019.
Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krähenbühl, Vladlen Koltun, and Dawn Song- Assessing generalization in deep reinforcement learning. arXiv preprint arXiv: 18...2282, 2018.
Aravind Rajeswaran, Kendall Lowrey, Emanuel V Todorov, and Sham M Kakade. Towards gen- eralization and simplicity in continuous control. In Advances in Neural Information Processing Systems, 2017.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interfer- ence. In International Conference on Learning Representations, 2018
Tom Schaul, Diana Borsa, Joseph Modayil, and Razvan Pascanu, Ray interference: a source of plateaus in deep reinforcement learning.
2019.
Richard S Sutton. Generalization in reinforcement learning: Successful examples using sparse coarse coding. In Advances in Neural Information Processing Systems, 1996.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations, 2019.
Ronald J Williams. Tight performance bounds on greedy policies based on imperfect value func- tions. Technical report, Citeseer, 1993.
10

---

Under review as a conference paper at ICLR 2021
A EXPERIMENTAL DETAILS
A.1 EXPERIMENT SETUP
We experiment with three environments: CartPole, Acrobot and Mountain Car from the OpenAl gym (https: hpt.gp.............mmmmmmm com/). We set the maximum steps per episode to 500, and the number of training iterations to 400. use a discounting factor y = 0.99 all environments. CartPole, Acrobot, and Mountain Car have 4, 6, and 2 dimensional states respectively. For tile coded inputs, we use 4 tiles and 8 tilings for each dimension of the state seperately, which results in input size of 160 and 240 respectively in CartPole and Acrobot.
We use 40 Monte Carlo rollouts to estimate the performance of the policy at each iteration a (Q7 a)). For evaluating the TD error difference, which approximates uniform sampling from all the past transition. For the experiment Section 5, Interference Across Iterations and Forgetting computed over the last 200 iterations.
reservoir buffer of size 1000,
A.2 NETWORK ARCHITECTURE AND HYPERPARAMETERS
For all experiments, initialize the neural networks, and Adam optimizer.
two-layer neural network with ReLU activation, He intialization to
For the experiments in Section 4, we generate set of hyper-parameter θ by choosing each param- eter in the set:
batch size 64
Step size 0.0003
buffer size {100, 5000, 10000}
Hidden size E (64, 128, 256, 512]
Number of step in an iteration M E (100,
Numver of iteration 400
Input type E {state, tile coded state}
For the experiments in Section 5, we sweep the hyperparameter in the range:
Batch size 64
Step size (0.001, 0.0003, 0.0001}
Hidden size 128
Buffer size E {1000, 5000, 1000}
The parameters are chosen based on average performance over the last 200 iterations.
A.3
ITERATIVE VALUE ESTIMATION ALGORITHMS WITH SEPARATE UPDATES
We provide a complete description of the algorithm in Algorithm
A.4 ONLINE-AWARE META-LEARKE LEARNING
This OML objective can be optimized similarly other gradient-based meta-learning objec- tives (Finn et al. 2017), and can be implemented in API and FQI. We provide a com- Note that the original algorithm from Javed and White (009 samples Bı...., B, in sequential manner, that is, B,...., B, (S1+1, 04, Ft+1, S 1)..)..... (St+1, aty Ftts, s(+), a trajectory from the buffer, but we sample each of .,.... B, as a random mini-batch of transitions from the replay buffer.
plete description of the algorithm in Algorithm
......,,
In our experiment, We sweep over the hyperparameters the set:
Inner update optimizer SGD

---

Under review as a conference paper at ICLR 2021
Meta update optimizer Optimizer for the last layer Adam
Q E {0.001, 0.0003, 0.0001}
Olinner E {0.01, 0.001, 0.0001, 0.00001]
Number of inner updates E {10, 20, 40)
A.5
SPARSE REPRESENTATION NEURAL NETWORKS
Liu et al. (2019) use the distributional regularizers to learn sparse representation neural networks (SRNN). If the vanilla neural network optimizes the objective J, SRNN simply adds regularization:
JSRNN w) J(3,w) + ASKL Σ SKLo(03s)
where SKL(Ø) is component-wise regularization E(e,))BB (s,a)) and Φ denote the j-th component of Φ. ÀSKL controls the weight on the reg- ularization and control the sparsity level. SRNN can be implemented API and FQI by simply adding regularization when updating the parameter
the expected activation Φ
In our experiment, we sweep over the key hyper-parameters of SRNN in the set:
Adam optimization
a E {0.001, 0.0003, 0.0001}
Äskl {0.1, 0.01, 0.001, 0.0001)
0 E (0.1,0.2,0.4)
B DISTRIBUTION OF ACCURACY CHANGE
In this experiment, we visualize the distribution of Accuracy Change over state-action pairs. We run the API-Stage algorithm on CartPole for 40k steps and choose three time steps corresponding to the time step (1) when the average Accuracy Change is the largest (2) when the average Accuracy Change is the smallest (3) when average Accuracy Change is the closest to
We show the distribution of Accuracy Change due to the internal layers and the last layer, for the three time step we choose, in Figure 5 The results suggests that updates in the internal layers has smaller magnitude of interference than updates in the last layer for all the three cases.
253
Interall Layer
200
132
200
Leat Layer
160
Feo www.soprra00p
Average is highly positive.
Average is highly negative.
Average is close zero.
Figure 5: Histogram of Accuracy Change in CartPole.
C ADDITIONAL EXPERIMENT RESULTS OF SECTION 6
We compare the algorithms mentioned in Section 6 in CartPole, Acrobot and Mountain Car, and show the learning curve in Figure 6 We found that OML significantly improve stability and effi- ciency compared to API-ORL and API-SRNN in Acrobot and Mountain Car. All algorithms per- form similarly in CartPole, possibly because they all choose the smallest step size the range in this domain.
12

---

Under review as a conference paper at ICLR 2021
100
-60
Return
Return
Retur
800
95
-80
API- OPI-ORL API-SRNN
API-OML
90
-100
-100
200
400
0
200
400
0
200
400
Number of Iterations
Number of Iterations
Number of Iterations
(a) CartPole
(b) Acrobot
(c) Mountain Car
Figure 6: Learning curve of API with different representation learning algorithms and FQI. The number are averaged over 30 runs with standard error.
Algorithm Iterative Value Estimation with separate updates
Initialize weights 0g. Initialize an empty buffer of size B. for t + 1.2....W do
If t mod Toval = then Qk < Qa., update Ta to be greedy w.r. Qs. by to be c-greedy Choose a, ~ by. (se), observe 1 and add the transition the buffer Sample mini-batch of transitions B, from the buffer and update the the internal layers fe-1 = 3r + a [B, E(sanaa) 83,, We: o, T, s') p. 9a,.w.
Update the the last layer
Wx+1 a T ((s.) 8(3r41, a,,,a, a, V Q8 (8,a) a)
Algorithm Iterative Value Estimation with online-aware representation learning
Initialize weights Qg. Initialize empty buffer of size B. for /. +- 1,2,... ... do
If t mod Toval = then Qk + Qo., update Ty to be greedy w.r.t Qs+ by to be e-greedy Choose a, ~ (x())). observe (st+1, Te+1), and add the transition to the buffer // Representation inner update:
Wi, Wt for i + 2, do
Sample transitions mini-batch B, from the buffer
We, f- Wi,s-1-Cinne [B. E(aane)) 8(3..
Sample mini-batch of B, from the buffer // Representation meta update:
3--1 = Br + a aT (sann))) 8(3,, W. a 11 Last layer update:
W We a Eiaand) 8(3,11, w.s. (9,
13


Reviews:
[{"review_id": "zeh6EzlSmIA", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes new measures to quantify interference in reinforcement learning at different granularity and show the correlation between interference and forgetting, and suggests to use extra representation loss to reduce interference. \n\nIn general, the definition of  interference at different granularity looks reasonable and the relation with forgetting matches intuition. However, there are no experiments to show any difference between those interference, such as update interference,  iteration interference, and interference across iterations.  So what's point to define interference with different granularity if only one is used in the end?  \n\nThe authors propose using extra representation loss to reduce interference, but the motivation is rather vague.  As the experiments show that internal layers contribute much less to interference, why it is better to adjust internal layers by representation loss than adjust the last layer which has much more influence on interference?  Is there more solid justification to connect representation loss and interference?  \n\nThe authors also introduces three different types of representation loss and OML obviously outperforms the others, but there is no enough analysis about why OML is supreme and then no guidance of how to choose a proper representation loss.\n\nAs I'm not an expert in reinforcement learning, I will leave the novelty and significance to other reviewers to decide. \n\nSome minor issues:\n1. Equations should be numbered.\n2. Could authors elaborate a bit on how to get the last equation on page 4? As the last term in the right side hasn't appeared before. \n3. The coefficients of forgetting and interference in the subfigures of Fig. 1 should be provided.\n \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "xjV-sP0ouSt", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the the interference problem under the API and FPI setting. It designs new measure for interference, and shows that the interference measure is correlated with forgetting. With the help of the interference measure, the paper studies the importance of the final layer of the neural network and proposes a new algorithm to mitigate interference. \n\nOverall, the paper is well-written but I have some questions regarding the logic of the paper. First, it's not super clear to me what is the role of the forgetting measure. Since the validation of interference measure is evaluated by forgetting, I'm assuming forgetting is a more commonly used measure (please correct me if I'm wrong). \n- If this is the case, then it seems the importance of interference measure is to come up with the SRNN and OML variants. But I don't get why we cannot use the forgetting measure for regularization, maybe some clarifications needed here. If the interference measure is not necessary for OML and SRNN, I don't quite get the necessarily of the interference measure. \n- If this is not the case, then why forgetting is an interesting measure to compare with? \n\nIn addition, I think it may be easier to follow if the unused definitions can be moved to Appendix, e.g., Pointwise Interference and Interference Across Iterations are only mentioned when they're defined, On the other hand, I think it'll be helpful to move some contents from Appendix to Sec 6.2, since it is the place that explicitly take advantage of the interference measure. \n\nLastly, there two minor points: \n- Line 4. the definition of iid is missing\n- Algorithm 1: it seems the the definition of b_k is missing", "extra_scores": {}, "extra_text": {}}, {"review_id": "NuMclkgiSnM", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThe paper studies interference and forgetting in the context of reinforcement learning (RL).  On the example of the Iterative Value Estimation family of algorithms, the authors define interference as the increase of the true Q target prediction error after updating Q function parameters. Since the true Q target is usually unknown, the authors propose to use the difference of squared TD-errors between updates as a proxy for interference. The paper further defines forgetting as the difference between the current agent performance and the best performance across all previous updates. On CartPole and Acrobot environments, the authors show a positive correlation between the proposed measures of interference and forgetting. They further qualitatively demonstrate that updates of the last layer weights result in higher interference compared to updates of intermediate layers. Finally, the authors propose an algorithm based on meta-learning for learning representations that minimize interference resulting in more stable return plots on Acrobot. \n\nStrengths\n- The topic brought in the paper is important: the distribution shift and moving targets in RL are indeed problematic and often lead to instabilities during training and forgetting of high-return policies.\n- The paper is written clearly and generally easy to follow.\n\nWeaknesses\n- The proposed algorithm is designed to explicitly minimize interference. However, seeking for minimizing interference / forgetting on its own might prohibit exploration. For example, an agent that achieves the worst possible returns and is not learning at all will have zero interference / forgetting. (Perhaps it is more reasonable to seek monotonic policy improvement?)\n- The finding that changes in parameters of the last layer result in higher interference seems unsurprising as, generally, changes in the final layer parameters affect the output of a neural network more than changes in intermediate layers.\n- The proposed method for learning representations is based on meta-learning. It is unclear whether the learning curves on Acrobot are more stable due to claimed minimization of interference or due to using a more powerful optimization method. An ablation study will improve the quality of the paper.\n- The authors should consider using harder environments to make the experimental results more convincing.\n\nPositioning relative to the literature\n- One of the main methodological contributions of the paper is using squared TD-error as a proxy for measuring interference. At the end of section 4, the authors mention related methods based on first-order approximation of interference. However, no comparison with the methods was provided.\n- Similarly, the authors do not compare with other methods that address the forgetting in the RL context. For example, [2] uses simple weight averaging for minimizing forgetting and achieves learning curves similar to the ones in Figure 4.\n- Perhaps the authors should reconsider the term “interference”. The cited paper [1] uses the term “interference” for an inner product between gradient estimates evaluated at different data points and seeks, in contrast, to maximize interference.\n\nRecommendation\n\nThe reviewer leans towards rejecting the paper. The discussed problem is important, however, the findings of the paper do not seem generally surprising. The authors claim that “there is no established online measure of interference for RL” but the resulting measure of interference is simply a difference between squared TD-errors before and after an update. The proposed measure of forgetting, being a difference of current returns and previous best returns, has limited novelty too. Moreover, the paper lacks a comparison with related work. Addressing the outlined weaknesses might increase the assigned score.\n\n**\n\nPost-rebuttal update: the score is increased from 4 to 5. See the response to the authors' comments.\n\n**\n\n[1] Bengio, Emmanuel, Joelle Pineau, and Doina Precup. \"Interference and Generalization in Temporal Difference Learning.\" arXiv preprint arXiv:2003.06350 (2020).\n\n[2] Nikishin, Evgenii, Pavel Izmailov, Ben Athiwaratkun, Dmitrii Podoprikhin, Timur Garipov, Pavel Shvechikov, Dmitry Vetrov, and Andrew Gordon Wilson. \"Improving stability in deep reinforcement learning with weight averaging.\" In Uncertainty in artificial intelligence workshop on uncertainty in Deep learning. 2018.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "7bxwste9euB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the reason for interference, aka catastrophic forgetting, when using parametric models for Reinforcement Learning. The authors draw the connection with previous methods and introduce some reasonable measure of interference. Then, they introduce a method to explicitly address the problem of interference, showing some good empirical results w.r.t. selected baselines.\n\nI think that this paper studies an interesting problem, but its analysis is a bit superficial and not supported by rigorous theoretical analysis. The proposed measures of interference make sense, but it seems to me that they are not reliable measures, especially in the considered online learning setting. In fact, the TD-error may increase significantly across several iterations because of the agent exploring unvisited states, and not only because of interference. Previous works, e.g. Prioritized Experience Replay (Schaul et al, 2016), show that a higher TD-error is actually desirable to guide exploration, so I'm unsure how the motivation behind this work relates with the literature. Moreover, since catastrophic forgetting is mostly problematic in deep RL, I'd have liked a stronger focus on deep RL, where the author could have tested the benefit of their approach on several algorithms based on the TD error, e.g. DQN, DDPG, TD3, SAC. It is also confusing how the authors refer to FQI as an online algorithm. FQI is known to be a batch RL algorithm, i.e. an offline algorithm where a fixed dataset of transitions collected by another agent is available, even though it can be used for iterative policy updates, but this is not the typical scenario of FQI.\n\nWithout an extensive empirical analysis, and considering the absence of a rigorous theoretical analysis motivating the proposed interference measure, I think this paper is not ready for publication and I encourage the authors to improve it, especially showing stronger and more significant empirical evidence over representative baselines, perhaps even considering some multi-task and/or lifelong learning problems where interference constitutes a bigger issue.\n\nPros\n------\n* The considered problem is interesting for a broad group of researchers in RL;\n* The presentation is clear enough.\n\nCons\n-------\n* The proposed measures are intuitive, but not supported by strong theoretical guarantees. In particular, the effect of exploration on the behavior of the TD error is not accurately discussed;\n* Absence of deep RL experiments to show the effectiveness of the proposed approach on more challenging problems.\n\n\nPost-rebuttal feedback\n-------------------------------\nI thank the authors for their reply. I still think that this paper has the major problem of presenting results about interference that are not strong enough to be published. In particular, I agree when the author say that \"the paper tries to bring conceptual clarity to this important topic, and provide a clear empirical methodology to measure interference and understand correlations to forgetting\", but still I think that the overall contribution consists \"just\" of one (of several other possible ones) intuitive method to measure interference, without a solid motivation. To me, this paper is promising but should present more significant results to have a stronger impact. I think the options are only two: stronger theoretical motivation, larger empirical analysis especially considering deep RL. Between the two, I consider the second option the best.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates interference in reinforcement learning and introduces a novel measure that can be used in value-based methods.
Although the reviewers acknowledge that the paper has merits (the topic is relevant and the paper is well written), they feel that the contribution is not sufficiently supported by either a theoretical or empirical analysis. The authors' responses have solved some of the reviewers' concerns, but they agree that this paper is not ready for publication in its current form.
I encourage the authors to update their paper following the reviewers' suggestions, in particular by improving the empirical analysis where comparisons with alternative methods (e.g., AVI/API methods that introduce regularization) need to be added.
Decision: Reject