Constructing a Good Behavior Basis for Transfer using Generalized Policy Updates

1 INTRODUCTION . Reinforcement learning ( RL ) studies the problem of building rational decision-making agents that maximize long term cumulative reward through trial-and-error interaction with a given environment . In recent years , RL algorithms combined with powerful function approximators such as deep neural networks have achieved significant successes in a wide range of challenging domains ( see e.g . Mnih et al . ( 2015 ) ; Vinyals et al . ( 2019 ) ; Silver et al . ( 2017 ; 2018 ) ) . However , these algorithms require substantial amounts of data for performing very narrowly-defined tasks . In addition to being datahungry , they are also very brittle to changes in the environment , such as changes in the tasks over time . The most important reasons behind these two shortcomings is that RL algorithms usually learn to perform a task from scratch , without leveraging any form of prior knowledge , and they are trained to optimize performance on only a single task . A promising approach to tackle both of these shortcomings is to learn multiple ways of behaving , i.e. , multiple policies that optimize different reward functions , and to reuse them as needed . Having access to multiple pre-learned policies can allow an agent to quickly solve reoccurring tasks in a lifelong RL setting . It can also allow for learning to combine the existing policies via a meta-policy to quickly learn new tasks , as in hierarchical RL . Recently , Barreto et al . ( 2020 ) have proposed the “ generalized policy updates ” framework , which generalizes the classical policy evaluation and policy improvement operations that underlie many of today ’ s RL algorithms . Its goal is to allow reusing policies resulting from previously learned tasks in order to perform well on downstream tasks , while also being data-efficient . More precisely , after learning the successor features of several policies in a policy set , also referred to as a behavior basis , they were able to instantaneously “ synthesize ” , in a zero-shot manner , new policies to solve downstream tasks , via generalized policy improvement . However , this work leaves open two important questions : ( i ) what set of policies should the agent learn so that its instantaneous performance on all possible downstream tasks is guaranteed to be good , and ( ii ) under what conditions does such a set of policies exist . In this paper , we provide answers to the questions above by proving that under certain assumptions about the environment dynamics and features , learning a diverse set of policies , which we call a set of independent policies , indeed guarantees good instantaneous performance on all possible downstream tasks . After providing an iterative algorithm for the construction of this set , we perform several experiments that validate our theoretical findings . In addition to the validation experiments , we compare this algorithm with recently proposed diverse policy set construction methods ( Eysenbach et al. , 2018 ; Zahavy et al. , 2020 ; 2021 ) and show that , unlike these methods , our approach is able to construct a behavior basis that enables instantaneous transfer to all possible tasks . We also show empirically that learning a set of independent policies can better bootstrap the learning process on downstream tasks where the reward function can not be described by a linear combination of the features . Finally , we demonstrate the usefulness of this set in a lifelong RL scenario , in which the agent faces different tasks over its lifetime . We hope that our study will bring the community a step closer to building lifelong RL agents that are able to perform multiple tasks and are able to instantaneously/quickly adapt to new ones during its lifetime . 2 BACKGROUND . Reinforcement Learning . In RL ( Sutton & Barto , 2018 ) , an agent interacts with its environment by choosing actions to get as much as cumulative long-term reward . The interaction between the agent and its environment is usually modeled as a Markov Decision Process ( MDP ) . An MDP is a tuple M ≡ ( S , A , P , r , d0 , γ ) , where S is the ( finite ) set of states , A is the ( finite ) set of actions , P : S × A × S → [ 0 , 1 ] is the transition distribution , r : S × A × S → R is the reward function , which specifies the task of interest , d0 : S → [ 0 , 1 ] is the initial state distribution and γ ∈ [ 0 , 1 ) is the discount factor . In RL , typically the agent does not have any knowledge about P and r beforehand , and its goal is to find , through pure interaction , a policy π : S → A that maximizes the expected sum of discounted rewards Eπ , P [ ∑∞ t=0 γ tr ( St , At , St+1 ) |S0 ∼ d0 ] , where Eπ , P [ · ] denotes the expectation over trajectories induced by π and P . Successor Features . The successor representation for a state s under a policy π allows s to be represented by the ( discounted ) distribution of states encountered when following π from s ( Dayan , 1993 ) . Given a policy , successor features ( SF , Barreto et al. , 2017 ) are a generalization of the idea of successor representations from the tabular setting to function approximation . Following Barreto et al . ( 2017 ) , we define SFs of a policy π for state-action ( s , a ) as : ψπ ( s , a ) ≡ Eπ , P [ ∞∑ i=0 γiφ ( St+i , At+i , St+i+1 ) ∣∣∣St = s , At = a ] , ( 1 ) where the ith component of ψπ gives the expected discounted sum of the ith component of the feature vector , φi , when following policy π , starting from the state-action pair ( s , a ) . Successor features allow a decoupling between the reward function and the environment dynamics . More concretely , if the reward function for a task can be represented as a linear combination of a feature vector φ ( s , a , s′ ) ∈ Rn : rw ( s , a , s ′ ) = φ ( s , a , s′ ) > ·w , ( 2 ) where w ∈ Rn , then , as we will detail below , the state-action value function Qπrw can be computed immediately as the dot-product of ψπ and w. The elements of w , wi , can be viewed as indicating a “ preference ” towards each of the features . Thus , we refer to w interchangeably as either the preference vector or the task . Intuitively , the elements φi of the feature vector φ can be viewed as salient events that maybe desirable or undesirable to the agent , such as picking up or leaving objects of certain type , and reaching and/or avoiding certain states . Generalized Policy Evaluation and Improvement . Generalized Policy Evaluation ( GPE ) and Generalized Policy Improvement ( GPI ) , together referred to as Generalized Policy Updates , are generalizations of the well-known policy evaluation and policy improvement operations in standard dynamic programming to a set of tasks and a set of policies ( Barreto et al. , 2020 ) . They are used as a transfer mechanism in RL to quickly construct a solution for a newly given task . One particularly efficient instantiation of GPE & GPI is through the use of SFs and value-based action selection . More concretely , given a set of MDPs having the following form : Mφ ≡ { ( S , A , P , rw = φ ·w , d0 , γ ) |w ∈ Rn } , ( 3 ) and given SFs ψπ ( s , a ) of a policy π , an efficient form of GPE on the task rw for policy π can be performed as follows : ψπ ( s , a ) > ·w = Qπrw ( s , a ) , ( 4 ) where Qπrw ( s , a ) is the state-action value function of π on the task rw . And , after performing GPE for all the policies π in a finite policy set Π , following Barreto et al . ( 2017 ) , an efficient form of GPI can be performed as follows : πGPIΠ ( s ) ∈ arg max a∈A max π∈Π Qπrw ( s , a ) . ( 5 ) We will refer to this specific use of SFs and value-based action selection for performing GPE & GPI as simply GPE & GPI throughout the rest of this study . Note that πGPI will in general outperform all the policies in Π , and that the actions selected by πGPI on a state may not coincide with any of the actions selected by π ∈ Π on that state . Hence , the policy space that can be attained by GPI can in principle be a lot larger than , e.g. , the space accessible by calling policies sequentially from the original set . 3 PROBLEM FORMULATION AND THEORETICAL ANALYSIS . GPE & GPI provide a guarantee that , for any reward function linear in the features , πGPI is at least as good as any of the policies π from the “ base set ” which was used to construct it . While this is an appealing guarantee of monotonic improvement , it does not say much , for two reasons . First , it is not clear how big an improvement can be expected for different tasks . More importantly , it leaves open the question of how one should choose base policies in order to ensure as much improvement as possible . After all , if we had a weak set of policies and we simply matched their value with πGPI , this may not be very useful . We will now show that , under certain assumptions , having access to a specific set of diverse policies , which we call a set of independent policies , can allow for instantaneously achieving high-level performance on all possible downstream tasks . Let us start by assuming that we are interested in a set of MDPsMφ , as defined in ( 3 ) , with deterministic transition functions ( the reason for the determinism assumption will become clear by the end of this section ) . For convenience , we also restrict the possible w values from Rn toW , where W is the surface of the ` 2 n-dimensional ball . Note that this choice does not alter the optimal policies of the MDPs inMφ , as an optimal policy is invariant with respect to the scale of the reward and W contains all possible directions . Next , we assume that the features φi that make up the feature vectors φ form a set of independent features ( SIF ) , defined as follows : Definition 1 ( SIF ) . A set of features Φ = { φi|φi : S × A × S → { 0 , C } , C ∈ R+ } ni=1 is called independent if , for any feature φi ∈ Φ and any initial state s0 ∼ d0 , we have : ( i ) φi ( s0 , a0 , s1 ) = 0 ∀a0 ∈ A and ∀s1 ∼ P ( s0 , a0 , · ) , and ( ii ) there exists at least one trajectory , starting from s0 , in which all the states associated with φi ( st , at , st+1 ) = C are visited , while the states associated with φj ( st , at , st+1 ) = C , ∀j 6= i , are not visited . It should be noted that a specific instantiation of this definition is the case where each feature is set to a positive constant at certain independently reachable state/states and zero elsewhere , which is the most common instantiation of feature vectors used in previous related work ( Barreto et al. , 2017 ; 2020 ) . We define the performance of an arbitrary policy π on a task rw as : Jπrw ≡ Eπ , P [ ∞∑ t=0 rw ( St , At , St+1 ) ∣∣∣S0 ∼ d0 ] , ( 6 ) whereEπ , P [ · ] denotes the expectation over trajectories induced by π and P . Note that Jπrw is a scalar corresponding to the expected undiscounted return of policy π under the initial state distribution d0 , which is the expected total reward obtained by π when starting from s0 ∼ d0 . We are now ready to formalize the problem we want to tackle : Problem Formulation . Given a set of MDPsMφ with deterministic transition functions and a SIF , we want to construct a set of n policies Πn = { πi } ni=1 , such that the performance of the policy πGPIΠn obtained by performing GPI on that set will maximize ( 6 ) for all rewards rw , where w ∈ W . That is , we want to solve the following problem : arg max Πn⊆Π J πGPIΠn rw for all w ∈ W. ( 7 ) It should be noted that the performance measure provided in ( 6 ) only measures the expected total reward and thus can not capture the optimality of the GPI policy . For instance , this measure can not distinguish between two policies that achieve the same expected total reward in a different number of time steps . However , Theorem 2 in Barreto et al . ( 2017 ) implies that , in general , the only way to guarantee the optimality of the GPI policy is to construct a behavior basis that contains all possible policies induced by all w ∈ W . Since there are infinitely many w values , this is impractical . Thus , in this study , we only consider GPI policies that maximize the expected total reward ( 6 ) . As a solution candidate to the problem in ( 7 ) , we now focus on a specific set of deterministic policies , called set of independent policies ( SIP ) , that are able to obtain features independently of each other : Definition 2 ( SIP ) . Let Φ = { φi } ni=1 be a SIF and let Π = { πi } ni=1 be a set of deterministic policies that are induced by each of the features in Φ. Π is defined to be a SIP if its elements , πi , satisfy : φj ( st , at , st+1 ) = φj ( s0 , a0 , s1 ) ∀j 6= i , ∀i , ∀s0 ∼ d0 and ∀t ∈ { 1 , . . . , T } , ( 8 ) where T is the horizon in episodic environments and T → ∞ in non-episodic ones , a0 = πi ( s0 ) and ( st , at , st+1 ) Tt=1 is the sequence of state-action-state triples generated by πi ’ s interaction with the environment . In general , having a SIP in a set of MDPs with stochastic transition functions is not possible , as the stochasticity can prevent ( 8 ) from holding . Thus , the assumption of a set of MDPs with deterministic transition functions is critical to our analysis . An immediate consequence of having this set of policies is that the corresponding SFs can be expressed in a simpler form , as follows : Lemma 1 . Let Φ be a SIF and let πi be a policy that is induced by the feature φi ∈ Φ and is a member of a SIP Π . Then , the entries of the SF ψπi of policy πi has the following form : ψπij ( s , a ) = { ψπii ( s , a ) , if i = j 0 , otherwise . ( 9 ) Due to the space constraints , we provide all the proofs in the supp . material . Lemma 1 implies that once we have a SIP , the SFs take the much simpler form ( 9 ) , which can allow for the GPI policy to maximize performance on all possible tasks according to the measure in ( 6 ) , solving the optimization objective in ( 7 ) : Theorem 1 . Let Φ be a SIF and let Π be a SIP induced by each of the features in Φ . Then , the GPI policy πGPIΠ is a solution to the optimization problem defined in ( 7 ) . Theorem 1 implies that having access to a SIP which consists of only n policies , where n is the dimensionality of φ , and applying the policy composition operator GPI on top , allows for instantaneously achieving maximum performance across all possible downstream tasks . Considering the fact that there are infinitely many such tasks , this provides a significant gain in the number of policies that are required to be learned for full downstream task coverage .

Summary: The paper extends the successor features framework to answer the following question: which policies should we learn and store so that, when presented with a new task, we achieve the best performance possible? The paper defines the notion of independent policies (forming a kind of basis over policy space), which can then be combined to solve new tasks (whose rewards are expressible as a linear combination of features) immediately. Experimental results support the theoretical results and show that it is best to learn these independent policies if we wish to maximise performance on downstream tasks.