Measuring and mitigating interference in reinforcement learning

1 INTRODUCTION . Generalization is a key property of reinforcement learning ( RL ) algorithms with function approximation . An agent must correctly generalize its recent experience to both states it has not yet encountered and other states it encountered in the past . Generalization has been extensively studied in supervised learning , inputs are sampled iid from a fixed input distribution and the targets are sampled from a fixed conditional distribution . The distribution of training data is often not iid . When learning from a stream of temporally correlated data , as in RL , the learner might fit the learned function to recent data and potentially overwrite previous learning—for example , the estimated values . This phenomenon is commonly called interference or forgetting in RL ( Bengio et al. , 2020 ; Goodrich , 2015 ; Liu et al. , 2019 ; Kirkpatrick et al. , 2017 ; Riemer et al. , 2018 ) . The conventional wisdom is that interference is particularly problematic in RL , even single-task RL , because ( a ) when an agent explores , it processes a sequence of observations , which are likely to be temporally correlated ; ( b ) the agent continually changes its policy , changing the distribution of samples over time ; and ( c ) most algorithms use bootstrap targets ( as in temporal difference learning ) , making the update targets non-stationary . It is difficult to verify this conventional wisdom , as there is no established online measure of interference for RL . There has been significant progress quantifying interference in supervised learning ( Chaudhry et al. , 2018 ; Fort et al. , 2019 ; Kemker et al. , 2018 ; Riemer et al. , 2018 ) , with some empirical work even correlating interference and properties of task sequences ( Nguyen et al. , 2019 ) , and investigations into ( un ) forgettable examples in classification ( Toneva et al. , 2019 ) . In RL , recent efforts have focused on generalization and transfer , rather than characterizing or measuring interference . Learning on new environments often results in drops in performance on previously learned environments ( Farebrother et al. , 2018 ; Packer et al. , 2018 ; Rajeswaran et al. , 2017 ; Cobbe et al. , 2018 ) . DQN-based agents can hit performance plateaus in Atari , presumably due to interference . In fact , if the learning process is segmented in the right way , the interference can be more precisely characterized with TD errors across different game contexts ( Fedus et al. , 2020 ) . Unfortunately this analysis can not be done online as learning progresses . Finally , recent work investigated several different possible measures of interference , but did not land on a clear measure ( Bengio et al. , 2020 ) . In this paper we advocate for a simpler approach to charactering interference in RL . In most systems the value estimates and actions change on every time-step conflating many different sources of non-stationarity , stochasticity , and error . If an update to the value function interferes , the result of that updated might not manifest in the policy ’ s performance for several time steps , if at all . Interference classically refers to an update negatively impacting the agent ’ s previous learning—eroding the agent ’ s knowledge stored in the value function . Therefore it makes sense to first characterize interference in the value function updates , instead of the policy or return . We define interference in terms of prediction error for two common approximate dynamic programming algorithms , approximate policy iteration and fitted Q iteration . Most value-based deep RL algorithms are based on these two algorithms . The interference is defined as the change in prediction errors , which is similar to previous definitions of interference in supervised learning . Additionally , our approach yields an online estimate of interference , which can even be directly optimized . In this work , we provide a clear justification for the use of differences in squared TD errors as the definition of interference . We highlight the definitions of interference at different granularities , and the utility of considering different statistics to summarize interference within iterations versus over time . We evaluate our interference measure by computing the correlation to a forgetting metric , which reflects instability in control performance . We show that high interference correlates with forgetting , and simultaneously show interference and forgetting properties across a variety of architectures and optimization choices . We then use our measure to highlight that updates to internal layers of the network—the representation—contribute much less to interference than updates on the last layer . This motivates the design of a new algorithm that learns representation online that explicitly minimize interference . We conclude with a demonstration that this algorithm does indeed significantly improve stability and reduce interference . 2 PROBLEM FORMULATION AND LEARNING ALGORITHMS . In reinforcement learning ( RL ) , an agent interacts with its environment , receiving observations and selecting actions to maximize a reward signal . We assume the environment can be formalized as a Markov decision process ( MDP ) . An MDP is a tuple ( S , A , Pr , R , γ , d0 ) where S is a set of states , A is an set of actions , Pr : S × A × S → [ 0 , 1 ] is the transition probability , R : S × A × S → R is the reward function , γ ∈ [ 0 , 1 ] a discount factor , and d0 is the initial distribution . The goal of the agent is to find a policy π : S ×A → [ 0 , 1 ] to maximize the expected discounted sum of rewards . Given a fixed policy π , the action-value function Qπ : S × A → R is defined as Qπ ( s , a ) : = E [ ∑∞ k=0 γ kRt+k+1|St = s , At = a ] , where Rt+1 = R ( St , At , St+1 ) , St+1 ∼ Pr ( ·|St , At ) , and actions are taken according to policy π : At ∼ π ( ·|St ) . Given a policy π , the value function can be obtained using the Bellman operator for action values T π : R|S|×|A| → R|S|×|A| : ( T πQ ) ( s , a ) : = ∑ s′∈S Pr ( s ′|s , a ) [ R ( s , a , s′ ) + γ ∑ a′∈A π ( a ′|s′ ) Q ( s′ , a′ ) ] . Qπ is the unique solution of the Bellman equation T πQ = Q . The optimal value function Q∗ is defined as Q∗ ( s , a ) : = supπ Q ( s , a ) , with π ∗ the policy that is greedy w.r.t . Q∗ . Similarly , the optimal value function can be obtained using the Bellman optimality operator for action values T : R|S|×|A| → R|S|×|A| : ( T Q ) ( s , a ) : =∑ s′∈S Pr ( s ′|s , a ) [ R ( s , a , s′ ) + γmaxa′∈AQ ( s′ , a′ ) ] . Q∗ is the unique solution of the Bellman equation T Q = Q . We can use neural networks to learn an approximation Qθ to the optimal action-value , with parameters θ . In this work , we restrict our attention to Iterative Value Estimation algorithms . These are algorithms where there is an explicit evaluation phase with a fixed policy , where the agent has several steps Teval to improve its value estimates . Two examples of such algorithms are Approximate Policy Iteration ( API ) and Fitted Q-Iteration ( FQI ) ( Ernst et al. , 2005 ) . In API , for the current policy πk , the agent updates its estimate of Qπk by taking Teval steps in the environment and performing a mini-batch update from a replay buffer on each step using the Sarsa update : θt+1 ← θt + αδt∇θtQθt ( St , At ) where δt : = Rt+1 + γQθt ( St+1 , πk ( St+1 ) ) −Qθt ( St , At ) . In FQI , the policy and targets Qk are held fixed for Teval steps , with these fixed targets used as a regression target in the update . Again , a mini-batch update update from a replay buffer is used on each step as above , but with a different δ = Rt+1 + γmaxa′∈AQk ( St+1 , a′ ) − Qθt ( St , At ) . The procedure for both algorithms is summarized in Algorithm 1 . Algorithm 1 Iterative Value Estimation : A General Framework for API and FQI Initialize weights θ0 . Initialize an empty buffer of size B. for t← 0 , 1 , 2 , . . . do If t mod Teval = 0 then Qk ← Qθt , update πk to be greedy w.r.t Qk , bk to be -greedy Choose at ∼ bk ( st ) , observe ( st+1 , rt+1 ) , and add the transition to the buffer Sample a mini-batch of transitions Bt from the buffer and update the weights : θt+1 ← θt + α 1|Bt| ∑ ( s , a , r , s′ ) ∈Bt δ ( θt ; s , a , r , s ′ ) ∇θQθt ( s , a ) where for API : δ ( θt ; s , a , r , s′ ) = r + γQθt ( s ′ , πk ( s ′ ) ) ) −Qθt ( s , a ) and for FQI : δ ( θt ; s , a , r , s′ ) = r + γmaxa′ Qk ( s′ , a′ ) −Qθt ( s , a ) 3 DEFINING INTERFERENCE IN VALUE ESTIMATION . In this section , we define interference for Iterative Value Estimation algorithms . Because these methods have an evaluation phase which corresponds to one iteration , we can more clearly define interference within one iteration . We discuss interference at four different levels of granularity . Within each iteration—in each evaluation phase—we can ask : did the agent ’ s knowledge about its value estimates improve or degrade ? The evaluation phase is more similar to a standard prediction problem , where the goal is simply to improve the estimates of the action-values towards a clear target . Let f∗ be the target function , which is either f∗ ( s , a ) = Qπk ( s , a ) for API or f∗ ( s , a ) = E [ R+ maxa′ Qk ( S′ , a′ ) |S = s , A = a ] for FQI . Pointwise Interference At the most fine-grained , we can ask if an update , going from θt to θt+1 , resulted in interference for a specific point ( s , a ) . The change in accuracy at s , a after an update is Accuracy Change ( ( s , a ) , θt , θt+1 ) : = ( f∗ ( s , a ) −Qθt+1 ( s , a ) ) 2 − ( f∗ ( s , a ) −Qθt ( s , a ) ) 2 where if this number is negative it reflects that accuracy improved . This change resulted in interference if it is positive , and zero interference if it is negative , and so we have Pointwise Interference ( ( s , a ) , θt , θt+1 ) : = max ( Accuracy Change ( ( s , a ) , θt , θt+1 ) , 0 ) . Update Interference At a less fine-grained level , we can ask if the update generally improved our accuracy—our knowledge in our value estimates—across points . Update Interference ( θt , θt+1 ) : = max ( E ( S , A ) ∼d [ Accuracy Change ( ( S , A ) , θt , θt+1 ) ] , 0 ) where ( s , a ) are sampled according to distribution d , such as from a buffer of collected experience . Notice that this differs from the expected Pointwise Interference . There are settings where they could produce notably different values . For example , an agent could have high positive and negative Accuracy Change that cancel . The Update Interference reflects that , on average , the agent ’ s knowledge has not changed : it improved in some places , and degraded in others . The expected Pointwise Interference , on the other hand , would be high , because for some points interference was high . We focus first on Update Interference , since it is the coarser measure ; future work is to look in a more fine-grained way at Pointwise Interference . Both Pointwise Interference and Update Interference are about one step . At an even higher level , we can then ask how much interference we have across multiple steps , both within an iteration and across multiple iterations . At this higher level , it becomes more sensible to consider upper percentiles , to ask if there was significant interference within an iteration and across iterations . For this we take expectations over only the top α percentage of values . In finance , this is typically called the expected tail loss or conditional value at risk . Previous work in RL ( Chan et al. , 2020 ) has used conditional value at risk to measure the long-term risk of RL algorithms . Iteration Interference reflects if there was significant interference in updating during the evaluation phase ( an iteration ) . Even a few update steps having significant interference within an iteration could cause significant instability ; an average over the steps might wash out those few significant steps . We therefore define Iteration Interference for iteration k using expectation over the top 10 % of values Iteration Interference ( k ) : = E [ X|X ≥ Percentile0.9 ( X ) ] for X = Update Interference ( θT , k , θT+1 , k ) where T is the time step in the iteration k , uniformly distributed and Percentile0.9 ( X ) is the 0.9- percentile of the distribution of X . Other percentiles could be considered , where smaller percentiles average over more values and a percentile of 0.5 gives the median . Interference Across Iterations reflects if an agent has many iterations with significant interference . Once again , even a few iterations with significant interference could destabilize learning ; expectations over tails are more suitable than over all iterations . For iteration K a random variable , Interference Across Iterations : = E [ X|X ≥ Percentile0.9 ( X ) ] for X = Iteration Interference ( K ) These definitions are quite generic , assuming only that we have well-defined targets for the evaluation phase and an algorithm that proceeds in iterations . Though we have only discussed API and FQI , the algorithm DQN also fits well into this class—and so could be analyzed—because the use of target networks mimics FQI . The primary difference is that the policy changes within an iteration . This is not logistically an issue , as updates are off-policy from a replay buffer ; but , it does add an additional changing variable . We therefore focus our experiments on API and FQI .

Summary: The paper studies interference and forgetting in the context of reinforcement learning (RL).  On the example of the Iterative Value Estimation family of algorithms, the authors define interference as the increase of the true Q target prediction error after updating Q function parameters. Since the true Q target is usually unknown, the authors propose to use the difference of squared TD-errors between updates as a proxy for interference. The paper further defines forgetting as the difference between the current agent performance and the best performance across all previous updates. On CartPole and Acrobot environments, the authors show a positive correlation between the proposed measures of interference and forgetting. They further qualitatively demonstrate that updates of the last layer weights result in higher interference compared to updates of intermediate layers. Finally, the authors propose an algorithm based on meta-learning for learning representations that minimize interference resulting in more stable return plots on Acrobot.