Disentangled Cumulants Help Successor Representations Transfer to New Tasks

1 INTRODUCTION . Natural intelligence is able to solve many diverse tasks by transferring knowledge and skills from one task to another . For example , by knowing about objects and how to move them in 3D space , it is possible to learn how to sort them by shape or colour faster . However , many of the current state-of-the-art artificial reinforcement learning ( RL ) agents often struggle with such basic skill transfer . They are able to solve single tasks well , often beyond the ability of any natural intelligence ( Silver et al. , 2016 ; Mnih et al. , 2015 ; Jaderberg et al. , 2017 ) , however even small deviations from the task that the agent was trained on can result in catastrophic failures ( Lake et al. , 2016 ; Rusu et al. , 2016 ) . Although improving transfer in RL agents is an active area of research ( Higgins et al. , 2017a ; Rusu et al. , 2016 ; Nair et al. , 2018 ; Barreto et al. , 2018 ; Wulfmeier et al. , 2019 ; Torrey & Shavlik , 2010 ; Taylor & Stone , 2009 ; Thrun & Pratt , 2012 ; Caruana , 1997 ; Jaderberg et al. , 2017 ; Riedmiller et al. , 2018 ) , most typical deep RL agents start learning every task from scratch . This means that each time they have to re-learn how to perceive the world ( the mapping from a high-dimensional observation to state ) , and also how to act ( the mapping from state to action ) , with the majority of time arguably spent on the former . The optimisation procedure naturally discards information that is irrelevant to the task , which means that the learnt state representation is often unsuitable for new tasks . Biological intelligence appears to operate differently . A lot of knowledge tends to be discovered and learnt without explicit supervision ( Tolman , 1948 ; Clark , 2013 ; Friston , 2010 ) . This basic knowledge can then form the behavioural basis that can be used to solve new tasks faster . In this paper we argue that such transferable knowledge and skills should be acquired in artificial agents too . In particular , we want to start by building agents that have the ability to discover stable entities that make up the world and to learn basic skills to manipulate these entities . Compositional re-use of such skills enables biological intelligence to find reasonable solutions to many naturally occurring tasks , from goal-directed movement ( controlling your own position ) , to food gathering ( controlling the position of fruit and and nuts ) , or building a simple defence system ( re-positioning multiple stones into a fence or digging a ditch ) . In this paper we concentrate on goal-based natural tasks that can be expressed in natural language , and that do not require a specific execution order of actions . To this end , we propose a principled way to learn a small set of policies which can be re-used by the agents to quickly produce reasonable performance on an exponentially large set of goal-driven tasks within an environment . We propose a method on how to discover these policies in the absence of external supervision , where the agent accumulates a transferable set of basic skills through intrinsically motivated interactions with the environment . This first stage of free play builds the foundation to later solve many diverse extrinsically specified downstream tasks . We suggest formalising such a two-stage pipeline as the endogenous reinforcement learning ( ERL ) setting , in order to provide a consistent evaluation framework for some of the existing and future work on building RL agents with intrinsic learning signals ( Gregor et al. , 2017 ; Eysenbach et al. , 2019 ; Hansen et al. , 2019 ; Nair et al. , 2018 ; Laversanne-Finot et al. , 2018 ) . We propose a disjoint two step research pipeline , where the agent is allowed unlimited access to the environment in the ERL stage , where no extrinsic rewards are provided and the agent is supposed to learn as much as it can through endogenously ( intrinsically ) driven interactions with the environment . This is followed by a standard RL stage where the success of the previous step is evaluation in terms of data-efficiency of learning on multiple diverse extrinsically ( exogenously ) specified downstream tasks in the same environment . We hope that by working in this extreme two stage setting , where the agents have to learn useful knowledge with no access to task rewards , we can develop algorithms that learn more robust and transferable policies even in the traditional RL setting . In this paper we propose to use the ERL stage to discover disentangled features through task-free interactions with the environment , and then solving a number of goal-driven self-generated tasks specified in the learnt disentangled feature space . In particular , we suggest learning k disentangled features , discretising them intom bins each , and learning km feature control policies that achieve the respective bin value of the given feature . We then re-combine the feature control policies learnt in the ERL stage to solve downstream tasks in the RL stage in a few-shot manner using Generalized Policy Improvement ( GPI ) ( Barreto et al. , 2018 ) . We demonstrate theoretically and empirically that our proposed set of basis policies that learn to control disentangled features produce significantly better generalisation over a large number of downstream tasks . Intuitively , disentangled representations consist of the smallest set of features that represent those aspects of the world state that are independently affected by natural transformations and together explain the most of the variance observed in an environment ( Higgins et al. , 2018 ) ( see Fig . 1 ) . Disentangled features , therefore , carve the world at its joints and provide a parsimonious representation of the world state that also points to which aspects of the world are stable , and which can in principle be transformed independently of each other . We conjecture that disentangled features align well with the idealised state space in which natural tasks are defined . Hence , by learning a set of policies that can control these features an agent will acquire a set of basis policies which spans a large set of natural tasks defined in such an environment . Note that both disentangled features and their respective control policies can be learnt without an externally specified task , purely in the ERL setting . We provide both theoretical justification for this setup , as well as experimental illustrations of the benefit of disentangled representations in a large set of tasks of varying difficulty . Hence , the main contribution of this work is a theoretical result that extends the GPI framework to guarantee achievability on a large set of natural goal-driven tasks given a small set of basis policies that control disentangled features . In particular , we demonstrate that given k disentangled features discretised intom bins , we can guarantee achievability with a deterministic policy on at least ( m+1 ) k downstream tasks by using GPI to recombine km feature control policies discovered and learnt purely through intrinsically driven interactions with the environment in the total absence of environment rewards . Our result holds for any tasks that can be specified in natural language and do not require a particular ordering of the actions to be solved . For example , our approach would be able to solve a task that requires sorting objects in space based on their colour or shape , or tidying up a messy playroom by putting all the toys in a box , but it will not be able to solve a task like cooking a meal , where the execution order of the different stages in the recipe matters . Related work Past work on supervised and reinforcement learning has demonstrated how multitask learning , transfer and adaptation can provide strong performance gains across various domains ( Caruana , 1997 ; Thrun & Pratt , 2012 ; Yosinski et al. , 2014 ; Girshick et al. , 2014 ; Jaderberg et al. , 2017 ; Riedmiller et al. , 2018 ; Wulfmeier et al. , 2019 ) . Typically these approaches use hand-crafted auxiliary tasks to boost learning of the downstream tasks of interest , which is not scalable and comes with no guarantees on which set of auxiliary tasks is optimal for boosting performance on a large number of downstream tasks . A number of other past approaches shared our motivation of replacing the hand-crafted auxiliary task specification by an automatic way of discovering a diverse and useful set of policies in the absence of externally specified tasks . The predominant approach so far has been to optimise an objective that encourages behaviours that are both diverse and distinguishable from each other ( Gregor et al. , 2017 ; Eysenbach et al. , 2019 ; Hansen et al. , 2019 ) , or to learn how to solve intrinsic tasks sampled from a learnt representation space ( Nair et al. , 2018 ; Laversanne-Finot et al. , 2018 ) . While these approaches have been shown to be successful on transferring the learnt policies to solve certain downstream tasks , none of them provided theoretical guarantees on the downstream task coverage by the basis set of policies . Such guarantees were however provided by van Niekerk et al . ( 2018 ) and Barreto et al . ( 2017 ; 2018 ) . These papers calculated how well a given set of policies can be transferred to solve a wide range of downstream tasks . However , they left the question of how to discover such a set of basis policies open . Hence , our work provides a unique perspective by addressing both the questions of what makes a good basis set of policies to get certain guarantees on final task coverage , and how these policies may be learnt in the ERL setting . Other related literature worth noting is the work by Higgins et al . ( 2017b ) , who showed that learning a downstream task policy over disentangled representations improved its robustness to visual changes in the environment . Another piece of work ( Machado et al. , 2018 ) demonstrated the usefulness of discovering reward-agnostic options through successor feature learning for improving data efficiency in downstream task learning . The benefits of these options , however , were primarily through improving exploration . No guarantees were given in terms of downstream task coverage . 2 BACKGROUND . Basic Reinforcement Learning ( RL ) formalism . An RL agent interacts with its environment through a sequence of actions in such a way as to maximise the expected cumulative discounted rewards ( Sutton & Barto , 1998 ) . The RL problem is typically expressed using the formalism of Markov Decision Processes ( MDPs ) ( Puterman , 1994 ) . An MDP is a tuple M = ( S , A , P , R , γ ) , where S and A are the sets of states and actions , P is the transition probability that predicts the distribution over next states given the current state and action s′ ∼P ( ·|s , a ) , R is the distribution of rewards r∼R ( s , a , s′ ) received for making the transition s a7→ s′ , and γ ∈ [ 0,1 ) is the discount factor used to make future rewards progressively less valuable . Given an MDP , the goal of the agent is to maximise the expected returnGt= ∑∞ i=0γ irt+i . This is done by learning a policy π ( a|s ) that selects the optimal action a∈A in each state s∈S . A typical RL problem attempts to find the optimal policy π∗= argmax π E [ ∑ t≥0γ tr|π ] , where the expectation is taken over all possible interaction sequences of the agent ’ s policy with the environment . The optimal policy is learnt with respect to a particular task operationalised through the choice of the reward functionR ( s , a , s′ ) . Successor Features The successor feature ( SF ) representation is a way of decoupling the dynamics of an environment from its reward function . This is done by representing an environment reward as r ( s , a , s′ ) = φ ( s , a , s′ ) > w where φ ( s , a , s′ ) is a vector of environment features . Notably , this representation does not decrease the expressivity of r since no assumptions are made on the form of φ . Moreover , this representation allows for decomposing value functions as follows : Qπ ( s , a ) =Eπ [ ∞∑ k=0 γkφ ( st , at , st+1 ) |s0 =s , at=a ] > wj=ψ ( s , a ) π > wj . ( 1 ) where ψ ( s , a ) π is a vector of reward-independent successor features . GPI & GPE Generalised Policy Improvement ( GPI ) and Generalised Policy Evaluation ( GPE ) ( Barreto et al. , 2017 ) can be used to transfer a set of existing policies to solve new tasks . The framework is specified for a set of MDPs : Mφ ( S , A , P , γ ) = { Mφ ( S , A , P , r , γ ) | r ( s , a , s′ ) =φ ( s , a , s′ ) > w } ( 2 ) induced by all possible choices of weights w that specify all possible rewards r , given a state space S , action spaceA , transition probabilities P , discount factor γ and features φ ( s , a , s′ ) . Note that the features are meant to be the same for all MDPs M ∈Mφ . Given a policy πi learnt to solve task i specified bywi , we can evaluate its value under a different reward rj=φ ( s , a , s′ ) > wj using GPE : Qπij ( s , a ) =ψ ( s , a ) πi > wj ( 3 ) using our definition of successor features defined in ( 1 ) . Hence , given a set of policies π1 , π2 , ... , πi induced by rewards r1 , r2 , ... , ri over a subset of the MDPs M ′ ⊂Mφ , we can get a new policy πj for a new task induced by rj ( note that Mj ∈Mφ , Mj∩M ′=∅ ) according to : πj ( s ) =argmax a max i ψ ( s , a ) πi > wj . ( 4 )

Summary: The paper addresses the problem of policy transfer in reinforcement learning, which is an extremely relevant open problem in RL, and is being actively studied by the community.  The authors propose a framework for discovering a set of policies without external supervision which can then be used to produce reasonable performance on extrinsic tasks.  The work exhibits originality in that it shows that disentangled representations, learned by intrinsic rewards,  can lead to learn behaviours that are transferable to novel situations.