Unicorn: Continual learning with a universal, off-policy agent

1 INTRODUCTION . Continual learning , that is , learning from experience about a continuing stream of tasks in a way that exploits previously acquired knowledge or skills , has been a longstanding challenge to the field of AI ( Ring , 1994 ; Schaul , 2018 ) . A major draw of the setup is its potential for a fully autonomous agent to incrementally build its competence and solve the challenges that a rich and complex environment presents to it – without the intervention of a human that provides datasets , task boundaries , or reward shaping . Instead , the agent considers tasks that continually grow in complexity , as the agent ’ s competence increases . An ideal continual learning agent should be able to ( A ) solve multiple tasks , ( B ) exhibit synergies when tasks are related , and ( C ) cope with deep dependency structures among tasks ( e.g. , a lock can only be unlocked after its key has been picked up ) . A plethora of continual learning techniques exist in the supervised learning setting , as recently reviewed by Parisi et al . ( 2018 ) , who note that more research is needed to tackle continual learning with autonomous agents in uncertain environments - a setting well-suited to Reinforcement Learning ( RL ) . Previous work on continual learning with RL , and specifically on solving tasks with deep dependency structures , has typically focused on separating learning into two stages ; that is , first individual skills are acquired separately and then later recombined to help solve a more challenging task ( Tessler et al. , 2017 ; Oh et al. , 2017 ) . Finn et al . ( 2017 ) framed this as meta-learning and trained an agent on a distribution of tasks that is explicitly designed to adapt to a new task in that distribution with very little additional learning . In a linear RL setting Ammar et al . ( 2015 ) learn a latent basis via policy gradients to solve new tasks as they are encountered . Brunskill & Li ( 2014 ) derive sample complexity bounds for option discovery in a lifelong learning setting . In this work we aim to solve tasks with deep dependency structures using single-stage end-to-end learning . In addition , we aim to train the agent on all tasks simultaneously regardless of their complexity . We argue that , in order to tackle this problem , we need two basic ingredients : task generalization and off-policy learning . This allows experience to be shared across tasks , enabling the agent to efficiently develop representations and competency for each task in parallel . There are a number of RL techniques that perform task generalization by incorporating tasks directly into the definition of the value function ( Sutton et al. , 2011 ; Dietterich , 2000 ; Kaelbling , 1993 ) . Universal Value Function Approximators ( UVFAs ; Schaul et al . ( 2015 ) ) are the latest iteration of these works that effectively combines the value functions for multiple rewards and policies into a single function approximator . We extend UVFAs with off-policy learning ( Sutton et al. , 2011 ; Ashar , 1994 ; Peng & Williams , 1994 ) about multiple goals simultaneously , and scale them up to a parallel agent architecture ( Mnih et al. , 2016 ; Espeholt et al. , 2018 ) and train them end-to-end ( they were originally trained in a two-stage process ) . The resulting continual learning agent , called Unicorn1 , is capable of consistently solving continual learning tasks with deep dependency structures , at scale , in complex domains ( Figure 1 ) . It does this by sharing experience and reusing representations and skills across tasks . We claim the following contributions : • Unicorn : The first effective end-to-end training of UVFA at scale , off-policy , using a novel combination of state-of-the-art RL architectures . • Adapting the parallelized policy-based state-of-the-art RL architecture ( Espeholt et al. , 2018 ) to value-based as well as incorporating off-policy learning and off-policy corrections into the architecture . • A detailed investigation with multiple ablation experiments , showing that Unicorn effectively learns multiple tasks in parallel and exhibits synergies when tasks are related . • A continual learning experiment whereby the Unicorn learns to solve tasks with deep dependency chains ( e.g. , collect a key , unlock a lock , open a door , and collect a chest in that order . Then receive a reward upon completion of the task ) . This is achieved using task generalization and off-policy learning , to solve this task which was impossible for existing methods ( see the ‘ expert ( chest ) ’ baseline in Figure 1 , bottom ) . • Treasure world , a new continual learning domain with deep dependency structure that retains the full complexity of Beattie et al . ( 2016 ) ’ s DM Lab ( action space , vision , memory , navigation ) . This is especially important since continual learning benchmarks for RL are scarce ( Schaul , 2018 ) . 2 BACKGROUND . Reinforcement learning ( RL ) is a computational framework for making decisions under uncertainty in sequential-time decision making problems ( Sutton & Barto , 1998 ) . An RL problem is formulated as a Markov decision process , defined as a 5-tuple 〈S , A , r , P , γ〉where S is a set of states , A is a set of actions , r : S ×A → R is the reward function , P : S ×A×S → [ 0 , 1 ] is a transition probability distribution and γ ∈ [ 0 , 1 ) is a discount factor . A policy π maps states s ∈ S to a probability distribution over actions . We define the return from a given time step t as the discounted sum of rewards : Rt = rt+1 + γrt+2 + γ2rt+3 + · · · = ∑∞ k=0 γ krt+k+1 , where rt = r ( st , at ) . Action value functions Qπ ( s , a ) = Eπ [ Rt|st = s , at = a ] estimate the expected return for an agent that selects an action a ∈ A in some state s ∈ S , and follows policy π thereafter . The optimal action value function Q∗ ( s , a ) estimates the expected return with respect to the optimal policy π∗ . Q-learning ( Watkins , 1989 ; Watkins & Dayan , 1992 ) can be used to estimate the optimal action value function Q∗ ( s , a ) , via an iterative bootstrapping procedure in which Q ( st , at ) is updated towards a bootstrap target Zt that is constructed using the estimated Q-value at the next state : Zt = rt+1 + γmaxaQ ( st+1 , a ) . The difference δt = Zt −Q ( st , at ) is referred to as the temporal difference ( TD ) error ( Sutton & Barto , 1998 ) . Multi-step Q-learning variants ( Watkins , 1989 ; Peng & Williams , 1994 ; De Asis et al. , 2017 ) use multiple transitions in a single bootstrap target . A common choice is the n-step return defined as G ( n ) t = ∑n k=1 γ k−1rt+k + γ nmaxaQ ( st+n , a ) . In calculating n-step returns there may be a mismatch in the action selection between the target and behavior policy within the n steps . In order to learn off-policy , this mismatch can be corrected using a variety of techniques ( Sutton et al. , 2009 ; Maei et al. , 2010 ; Precup et al. , 2001 ; Munos et al. , 2016 ) . We deal with off-policy corrections by truncating the returns whenever a non-greedy action is selected , as suggested by Watkins ( 1989 ) . Universal Value Function Approximators ( UVFA ) extend value functions to be conditional on a goal signal g ∈ G , with their function approximator ( such as a deep neural network ) sharing an internal , goal-independent representation of the state f ( s ) ( Schaul et al. , 2015 ) . As a result , a UVFA Q ( s , a ; g ) can compactly represent multiple policies by conditioning on any goal signal g and choosing actions greedily . UVFA ’ s have previously been implemented in a two-stage process 1Unicorn stands for “ UNIversal Continual Off-policy Reinforcement learNing ” . involving a matrix factorization step to learn embeddings and a separate multi-variate regression procedure . In contrast , the Unicorn learns Q ( s , a ; g ) end-to-end , in a joint parallel training setup with off-policy goal learning and corrections . Tasks vs. goals . For the purposes of this paper , we assign distinct meanings to the terms task ( τ ) and goal signal ( g ) . A goal signal modulates the behavior of an agent ( e.g. , as input to the UVFA ) . In contrast , a task defines a pseudo-reward rτ ( e.g. , rkey = 1 if a key was collected and 0 otherwise ) . During learning , a vector containing all pseudo-rewards is visible to the agent on each transition , even if it is pursuing one specific goal . Each experiment defines a discrete set of K tasks { τ1 , τ2 , . . . , τK } . In transfer experiments , tasks are split between K ′ training tasks and K − K ′ hold-out tasks . 3 UNICORN . This section introduces the Unicorn agent architecture with the following properties to facilitate continual learning . ( A ) : The agent should have the ability to simultaneously learn about multiple tasks , enabling domains where new tasks are continuously encountered . We use a joint parallel training setup with a single learner but many actors working on different tasks to accomplish this ( sections 3.2 and section 3.4 ) . ( B ) : As the agent accumulates more knowledge , we want it to generalize by reusing some of its knowledge to solve related tasks . This is accomplished by using a single UVFA to capture knowledge about all tasks , with a separation of goal-dependent and goalindependent representations to facilitate transfer ( section 3.1 ) . ( C ) : The agent should be effective in domains where tasks have a deep dependency structure . This is the most challenging aspect , but is enabled by off-policy learning from the experience across all tasks ( section 3.3 ) . For example , an actor pursuing the door goal will sometimes , upon opening a door , subsequently stumble upon a chest . To the door-pursuing actor this is an irrelevant ( non-rewarding ) event , but when learning about the chest task this same event is highly interesting as it is one of the rare non-zero reward transitions . We will next describe the various components that comprise the Unicorn . The full Unicorn algorithm can be found in Appendix A . 3.1 VALUE FUNCTION ARCHITECTURE . A key component of the Unicorn agent is a UVFA , which is an approximator , such as a neural network , that learns to approximate Q ( s , a ; g ) . The power of this approximator lies in its ability to be conditioned on a goal signal g. This enables the UVFA to learn about multiple tasks simultaneously where the tasks themselves may vary in their level of difficulty ( e.g. , tasks with deep dependencies ) . Our proposed UVFA architecture is depicted schematically in Figure 2 : the current frame of visual input is processed by a convolutional network ( CNN ; LeCun et al . ( 1998 ) ) , followed by a recurrent layer of long short-term memory ( LSTM ; Hochreiter & Schmidhuber ( 1997 ) ) . As in ( Espeholt et al. , 2018 ) the previous action and reward are part of the observation . The output of the LSTM is concatenated with an “ inventory stack ” , to be described in section 4.1 , to form a goal-independent representation of state f ( s ) . This vector is then concatenated with a goal signal g and further processed via two layers of a multi-layer perceptron ( MLP ) with ReLU non-linearities to produce the output vector of Q-values ( one for each possible action a ∈ A ) . The union of trainable parameters from all these components is denoted by θ . Further details about the networks and hyperparameters can be found in Appendix B .

Summary: The authors aim to do continual learning to solve dependent tasks using "single-stage end-to-end learning". The resulting "Unicorn" agent trains on all tasks simultaneously. The idea is to use multi-task "off-policy learning", which uses (old) trajectories (experience) from task A to help learning on a related task B. Authors further distinguish between goals (inputs to Q) and tasks (different reward functions). A goal might a color/shape of an object to pick up.