Continual Auxiliary Task Learning

1 Introduction . In never-ending learning systems , the agent often faces long periods of time when the external reward is uninformative . A smart agent should use this time to practice reaching subgoals , learning new skills , and refining model predictions . Later , the agent should use this prior learning to efficiently maximize external reward . The agent engages in this self-directed learning during times when the primary drives of the agent ( e.g. , hunger ) are satisfied . Other times , the agent might have to trade-off directly acting towards internal auxiliary learning objectives and taking actions that maximize reward . In this paper we investigate how an agent should select actions to balance the needs of several auxiliary learning objectives in a no-reward setting where no external reward is present . In particular , we assume the agent ’ s auxiliary objectives are to learn a diverse set of value functions corresponding to a set of fixed policies . Our solution at a high-level is straightforward . Each auxiliary value function is learned in parallel and off-policy , and the behavior selects actions to maximize learning progress . Prior work investigated similar questions in a state-less bandit like setting , where both off-policy learning and function approximation are not required [ Linke et al. , 2020 ] . Otherwise , the majority of prior work has focused on how the agent could make use of auxiliary learning objectives , not how behavior could be used to improve auxiliary task learning . Some work has looked at defining ( predictive ) features , such as successor features and a basis of policies [ Barreto et al. , 2018 , Borsa et al. , 2019 , Barreto et al. , 2020 , 2019 ] ; universal value function approximators [ Schaul et al. , 2015 ] ; and features based on value predictions [ Schaul and Ring , 2013 , Schlegel et al. , 2021 ] . The other focus has been exploration , using auxiliary learning objectives to generate bonuses to aid exploration on the main task [ Pathak et al. , 2017 , Stadie et al. , 2015 , Badia et al. , 2020 , Burda et al. , 2019 ] ; using a given set of policies in a call-return fashion for scheduled auxiliary control 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . [ Riedmiller et al. , 2018 ] ; and discovering subgoals in environments where it is difficult for the agent to reach particular parts of the state-action space [ Machado et al. , 2017 , Colas et al. , 2019 , Zhang et al. , 2020 , Andrychowicz et al. , 2017 , Pong et al. , 2019 ] . In all of these works , the behavior was either fixed or optimized for the main task . The problem of adapting the behavior to optimize many auxiliary predictions in the absence of external reward is sufficiently complex to merit study in isolation . It involves several inter-dependent learning mechanisms , multiple sources of non-stationarity , and high-variance due to off-policy updating . If we can not design learning systems that efficiently learn their auxiliary objectives in isolation , then the agent is unlikely to learn its auxiliary tasks while additionally balancing external reward maximization . Further , understanding how to efficiently learn a collection of auxiliary objectives is complementary to the goals of using those auxiliary objectives . It could amplify the auxiliary task effect in UNREAL [ Jaderberg et al. , 2017 ] , improve the efficiency and accuracy of learning successor features and universal value function approximators , and improve the quality of the sub-policies used in scheduled auxiliary control . It can also benefit the numerous systems that discover options , skills , and subgoals [ Gregor et al. , 2017 , Eysenbach et al. , 2019a , Veeriah et al. , 2019 , Pitis et al. , 2020 , Nair et al. , 2020 , Pertsch et al. , 2020 , Colas et al. , 2019 , Eysenbach et al. , 2019b ] , by providing improved algorithms to learn the resulting auxiliary tasks . For example , for multiple discovered subgoals , the agent can adapt its behavior to efficiently learn policies to reach each subgoal . In this paper we introduce an architecture for parallel auxiliary task learning . As the first such work to tackle this question in reinforcement learning with function approximation , numerous algorithmic challenges arise . We first formalize the problem of learning multiple predictions as a reinforcement learning problem , and highlight that the rewards for the behavior policy are inherently non-stationary due to changes in learning progress over time . We develop a strategy to use successor features to exploit the stationarity of the dynamics , whilst allowing for fast tracking of changes in the rewards , and prove that this separation provides a faster convergence rate than standard value function algorithms like temporal difference learning . We empirically show that this separation facilitates tracking both for prediction learners with non-stationary targets as well as the behavior . 2 Problem Formulation . We consider the multi-prediction problem , in which an agent continually interacts with an environment to obtain accurate predictions . This interaction is formalized as a Markov decision process ( MDP ) , defined by a set of states S , a set of actions A , and a transition probability function P ( s , a , s0 ) . The agent ’ s goal , when taking actions , is to gather data that is useful for learning N predictions , where each prediction corresponds to a general value function ( GVF ) [ Sutton et al. , 2011 ] . A GVF question is formalized as a three tuple ( ⇡ , , c ) , where the target is the expected return of the cumulant , defined by c : S ⇥A⇥ S ! R , when following policy ⇡ : S ⇥A ! [ 0 , 1 ] , discounted by : S ⇥A⇥ S ! [ 0 , 1 ] . More precisely , the target is the action-value Q ( s , a ) def = E⇡ [ Gt|St = s , At = a ] for Gt def = Ct+1 + t+1Gt+1 where Ct+1 def = c ( St , At , St+1 ) and t+1 def = ( St , At , St+1 ) . The extension of to transitions allows for a broader class of problems , including easily specifying termination , without complicating the theory [ White , 2017 ] . The expectation is under policy ⇡ , with transitions according to P . The prediction targets could also be state-value functions ; we assume the targets are action-values in this work to provide a unified discussion of successor features for both the GVF and behavior learners . At each time step , the agent produces N predictions , a Q̂ ( j ) t ( St , At ) for prediction j with true targets Q ( j ) t ( St , At ) . We assume the GVF question can change over time , and so Q can change with time . The goal is to have low error in the prediction , in terms of the root mean-squared value error ( RMSVE ) , under state-action weighting d : S ⇥A ! R : RMSVE ( Q̂ , Q ) def= sX s2S X a2A d ( s , a ) ( Q̂ ( s , a ) Q ( s , a ) ) 2 ( 1 ) The total error up to time step t , across all predictions , is TE def= Pt i=1 PN j=1 RMSVE ( Q̂ ( j ) i , Q ( j ) i ) . Behavior Task 1 Task 2 Task N Intrinsic Reward Weight Change St at e/ O bs er va tio n Agent ( S t+ 1 ) < latexit sha1_base64= '' OjSwn01Y1y62YFQEFg9u2D/6H18= '' > AAAB8HicbVBNSwMxEJ2tX7V+VT16CRahIpRdqeix6MVjRfsh7VKyabYNTbJLkhXK0l/hxYMiXv053vw3pu0etPXBwOO9GWbmBTFn2rjut5NbWV1b38hvFra2d3b3ivsHTR0litAGiXik2gHWlDNJG4YZTtuxolgEnLaC0c3Ubz1RpVkkH8w4pr7AA8lCRrCx0mP5vpeaM29y2iuW3Io7A1omXkZKkKHeK351+xFJBJWGcKx1x3Nj46dYGUY4nRS6iaYxJiM8oB1LJRZU++ns4Ak6sUofhZGyJQ2aqb8nUiy0HovAdgpshnrRm4r/eZ3EhFd+ymScGCrJfFGYcGQiNP0e9ZmixPCxJZgoZm9FZIgVJsZmVLAheIsvL5PmecWrVi7uqqXadRZHHo7gGMrgwSXU4Bbq0AACAp7hFd4c5bw4787HvDXnZDOH8AfO5w+hlo+n < /latexit > Rt+1 < latexit sha1_base64= '' V1A7ppgGUup3oxpxVAu7QIY7BKo= '' > AAAB7nicbVBNS8NAEJ34WetX1aOXxSIIQkmkoseiF49V7Ae0oWy2m3bpZhN2J0IJ/RFePCji1d/jzX/jts1BWx8MPN6bYWZekEhh0HW/nZXVtfWNzcJWcXtnd2+/dHDYNHGqGW+wWMa6HVDDpVC8gQIlbyea0yiQvBWMbqd+64lrI2L1iOOE+xEdKBEKRtFKrYdehufepFcquxV3BrJMvJyUIUe9V/rq9mOWRlwhk9SYjucm6GdUo2CST4rd1PCEshEd8I6likbc+Nns3Ak5tUqfhLG2pZDM1N8TGY2MGUeB7YwoDs2iNxX/8zophtd+JlSSIldsvihMJcGYTH8nfaE5Qzm2hDIt7K2EDammDG1CRRuCt/jyMmleVLxq5fK+Wq7d5HEU4BhO4Aw8uIIa3EEdGsBgBM/wCm9O4rw4787HvHXFyWeO4A+czx/XnY9B < /latexit > At+1 < latexit sha1_base64= '' kReBLeP7qxIcNi6r2Ca46gqAW68= '' > AAAB7nicbVBNS8NAEJ34WetX1aOXxSIIQkmkoseqF48V7Ae0oWy2m3bpZhN2J0IJ/RFePCji1d/jzX/jts1BWx8MPN6bYWZekEhh0HW/nZXVtfWNzcJWcXtnd2+/dHDYNHGqGW+wWMa6HVDDpVC8gQIlbyea0yiQvBWM7qZ+64lrI2L1iOOE+xEdKBEKRtFKrZtehufepFcquxV3BrJMvJyUIUe9V/rq9mOWRlwhk9SYjucm6GdUo2CST4rd1PCEshEd8I6likbc+Nns3Ak5tUqfhLG2pZDM1N8TGY2MGUeB7YwoDs2iNxX/8zophtd+JlSSIldsvihMJcGYTH8nfaE5Qzm2hDIt7K2EDammDG1CRRuCt/jyMmleVLxq5fKhWq7d5nEU4BhO4Aw8uIIa3EMdGsBgBM/wCm9O4rw4787HvHXFyWeO4A+czx+9c48w < /latexit > EnvironmentThe agent ’ s goal is to gather data and update its predictions to make TE small . This goal can itself be formalized as a reinforcement learning problem , by defining rewards for the behavior policy that depend on the agent ’ s predictions . Such rewards are often called intrinsic rewards . For example , if we could directly measure the RMSVE , one potential intrinsic reward would be the decrease in the RMSVE after taking action At from state St and transitioning to St+1 . This reflects the agent ’ s learning progress—how much it was able to learn—due to that new experience . The reward is high if the action generated data that resulted in substantial learning . While the RMSVE is the most direct measure of learning progress , it can not be calculated without the true values . Algorithm 1 Multi-Prediction Learning System Input : N GVF questions Initialize behavior policy parameters ✓0 and GVF learners w ( 1 ) 0 , . . . , w ( N ) 0 Obtain initial observation S0 for t = 0 , 1 , . . . do Choose action At according to ⇡✓t ( ·|St ) Observe next state vector St+1 and t+1 // Update predictions with new data for j = 1 to N do c c ( j ) ( St , At , St+1 ) ( j ) ( St , At , St+1 ) Update w ( j ) t with ( St , At , c , St+1 , ) // Compute intrinsic reward , update behavior Rt+1 PN j=1 kw ( j ) t+1 w ( j ) t k1 Update ✓t with ( St , At , Rt+1 , St+1 , t+1 ) Many intrinsic rewards have been considered to estimate the learning progress of predictions . A recent work provided a thorough survey of different options , as well as an empirical study [ Linke et al. , 2020 ] . Their conclusion was that , for reasonable prediction learners , simple learning progress measures—like the change in weights—were effective for producing effective data gathering . We rely on this conclusion here , and formalize the problem using the ` 1 norm on the change in weights . Other intrinsic rewards could be swapped into the framework ; but , because our focus is on the non-stationarity in the system and because empirically we found this weight-change intrinsic reward to be effective , we opt for this simple choice upfront . We provide the generic pseudocode for a multiprediction reinforcement learning system , in Algorithm 1 . Note that the behavior agent also has a separate transition-based , which enables us to encode both continuing and episodic problems . For example , the pseudo-termination for a GVF could be a particular state in the environment , such as a doorway . The discount for the GVF would be zero in that state , even though it is not a true terminal state ; the behavior discount t+1 would not be zero . 3 Non-stationarity Induced by Learning . On the surface , the multi-prediction problem outlined in the previous section is a relatively straightforward reinforcement learning problem . The behavior policy learns to maximize cumulative reward , and simultaneously learns predictions about its environment . Many RL systems incorporate prediction learning , either as auxiliary tasks or to learn a model . However , unlike standard RL problems , the rewards for the behavior are non-stationary when using intrinsic rewards , even under stationary dynamics . Further , the prediction problems themselves are non-stationary due to a changing behavior . To understand this more deeply , consider first the behavior rewards . On each time step , the predictions are updated . Progressively , they get more and more accurate . Imagine a scenario where they can become perfectly accurate , such as in the tabular setting with stationary cumulants . The behavior rewards are high in early learning , when predictions are inaccurate . As predictions become more and more accurate , the change in weights gets smaller until eventually the behavior rewards are near zero . This means that when the behavior revisits a state , the reward distribution has actually changed . More generally , in the function approximation setting , the behavior rewards will continue to change with time , not necessarily decay to zero . The prediction problems are also non-stationary for two reasons . First , the cumulants themselves might be non-stationary , even if the transition dynamics are stationary . For example , the cumulant could correspond to the amount of food in a location in the environment , that slowly gets depleted . Or , the cumulant could depend on a hidden variable , that makes the outcome appear non-stationary . Even with a stationary cumulant , the prediction learning problem can be non-stationary due to a changing behavior policy . As the behavior policy changes , the state distribution changes . Implicitly , when learning off-policy , the predictions are minimizing an objective weighted by the state visitation under the behavior policy . As the behavior changes , the underlying objective is actually changing , resulting in a non-stationary prediction problem . Though there has been some work on learning under non-stationarity in RL and bandits , none to our knowledge has addressed the multi-prediction setting in MDPs . There has been some work developing reinforcement learning algorithms for non-stationary MDPs , but largely for the tabular setting [ Sutton and Barto , 2018 , Da Silva et al. , 2006 , Abdallah and Kaisers , 2016 , Cheung et al. , 2020 ] or assuming periodic shifts [ Chandak et al. , 2020a , b , Padakandla et al. , 2020 ] . There has also been some work in the non-stationary multi-armed bandit setting [ Garivier and Moulines , 2008 , Koulouriotis and Xanthopoulos , 2008 , Besbes et al. , 2014 ] . The non-stationary rewards for the behavior , that decay over time , have been considered for the bandit setting , under rotting bandits [ Levine et al. , 2017 , Seznec et al. , 2019 ] ; these algorithms do not obviously extend to the RL setting .

Summary: The paper proposes a framework for continually learning multiple generalized value functions, where each value function is learned off-policy and the behaviour policy learns to maximize the learning progress. The paper identifies the non-stationarity of the action value function during the process of maximizing learning progress and proposes an algorithm that separately learns the immediate reward and the successor feature. The experimental results show that the proposed algorithm significantly improves sample efficiency and perfroms well with the use of replay.