Constructing a Good Behavior Basis for Transfer using Generalized Policy Updates

1 INTRODUCTION . Reinforcement learning ( RL ) studies the problem of building rational decision-making agents that maximize long term cumulative reward through trial-and-error interaction with a given environment . In recent years , RL algorithms combined with powerful function approximators such as deep neural networks have achieved significant successes in a wide range of challenging domains ( see e.g . Mnih et al . ( 2015 ) ; Vinyals et al . ( 2019 ) ; Silver et al . ( 2017 ; 2018 ) ) . However , these algorithms require substantial amounts of data for performing very narrowly-defined tasks . In addition to being datahungry , they are also very brittle to changes in the environment , such as changes in the tasks over time . The most important reasons behind these two shortcomings is that RL algorithms usually learn to perform a task from scratch , without leveraging any form of prior knowledge , and they are trained to optimize performance on only a single task . A promising approach to tackle both of these shortcomings is to learn multiple ways of behaving , i.e. , multiple policies that optimize different reward functions , and to reuse them as needed . Having access to multiple pre-learned policies can allow an agent to quickly solve reoccurring tasks in a lifelong RL setting . It can also allow for learning to combine the existing policies via a meta-policy to quickly learn new tasks , as in hierarchical RL . Recently , Barreto et al . ( 2020 ) have proposed the “ generalized policy updates ” framework , which generalizes the classical policy evaluation and policy improvement operations that underlie many of today ’ s RL algorithms . Its goal is to allow reusing policies resulting from previously learned tasks in order to perform well on downstream tasks , while also being data-efficient . More precisely , after learning the successor features of several policies in a policy set , also referred to as a behavior basis , they were able to instantaneously “ synthesize ” , in a zero-shot manner , new policies to solve downstream tasks , via generalized policy improvement . However , this work leaves open two important questions : ( i ) what set of policies should the agent learn so that its instantaneous performance on all possible downstream tasks is guaranteed to be good , and ( ii ) under what conditions does such a set of policies exist . In this paper , we provide answers to the questions above by proving that under certain assumptions about the environment dynamics and features , learning a diverse set of policies , which we call a set of independent policies , indeed guarantees good instantaneous performance on all possible downstream tasks . After providing an iterative algorithm for the construction of this set , we perform several experiments that validate our theoretical findings . In addition to the validation experiments , we compare this algorithm with recently proposed diverse policy set construction methods ( Eysenbach et al. , 2018 ; Zahavy et al. , 2020 ; 2021 ) and show that , unlike these methods , our approach is able to construct a behavior basis that enables instantaneous transfer to all possible tasks . We also show empirically that learning a set of independent policies can better bootstrap the learning process on downstream tasks where the reward function can not be described by a linear combination of the features . Finally , we demonstrate the usefulness of this set in a lifelong RL scenario , in which the agent faces different tasks over its lifetime . We hope that our study will bring the community a step closer to building lifelong RL agents that are able to perform multiple tasks and are able to instantaneously/quickly adapt to new ones during its lifetime . 2 BACKGROUND . Reinforcement Learning . In RL ( Sutton & Barto , 2018 ) , an agent interacts with its environment by choosing actions to get as much as cumulative long-term reward . The interaction between the agent and its environment is usually modeled as a Markov Decision Process ( MDP ) . An MDP is a tuple M ≡ ( S , A , P , r , d0 , γ ) , where S is the ( finite ) set of states , A is the ( finite ) set of actions , P : S × A × S → [ 0 , 1 ] is the transition distribution , r : S × A × S → R is the reward function , which specifies the task of interest , d0 : S → [ 0 , 1 ] is the initial state distribution and γ ∈ [ 0 , 1 ) is the discount factor . In RL , typically the agent does not have any knowledge about P and r beforehand , and its goal is to find , through pure interaction , a policy π : S → A that maximizes the expected sum of discounted rewards Eπ , P [ ∑∞ t=0 γ tr ( St , At , St+1 ) |S0 ∼ d0 ] , where Eπ , P [ · ] denotes the expectation over trajectories induced by π and P . Successor Features . The successor representation for a state s under a policy π allows s to be represented by the ( discounted ) distribution of states encountered when following π from s ( Dayan , 1993 ) . Given a policy , successor features ( SF , Barreto et al. , 2017 ) are a generalization of the idea of successor representations from the tabular setting to function approximation . Following Barreto et al . ( 2017 ) , we define SFs of a policy π for state-action ( s , a ) as : ψπ ( s , a ) ≡ Eπ , P [ ∞∑ i=0 γiφ ( St+i , At+i , St+i+1 ) ∣∣∣St = s , At = a ] , ( 1 ) where the ith component of ψπ gives the expected discounted sum of the ith component of the feature vector , φi , when following policy π , starting from the state-action pair ( s , a ) . Successor features allow a decoupling between the reward function and the environment dynamics . More concretely , if the reward function for a task can be represented as a linear combination of a feature vector φ ( s , a , s′ ) ∈ Rn : rw ( s , a , s ′ ) = φ ( s , a , s′ ) > ·w , ( 2 ) where w ∈ Rn , then , as we will detail below , the state-action value function Qπrw can be computed immediately as the dot-product of ψπ and w. The elements of w , wi , can be viewed as indicating a “ preference ” towards each of the features . Thus , we refer to w interchangeably as either the preference vector or the task . Intuitively , the elements φi of the feature vector φ can be viewed as salient events that maybe desirable or undesirable to the agent , such as picking up or leaving objects of certain type , and reaching and/or avoiding certain states . Generalized Policy Evaluation and Improvement . Generalized Policy Evaluation ( GPE ) and Generalized Policy Improvement ( GPI ) , together referred to as Generalized Policy Updates , are generalizations of the well-known policy evaluation and policy improvement operations in standard dynamic programming to a set of tasks and a set of policies ( Barreto et al. , 2020 ) . They are used as a transfer mechanism in RL to quickly construct a solution for a newly given task . One particularly efficient instantiation of GPE & GPI is through the use of SFs and value-based action selection . More concretely , given a set of MDPs having the following form : Mφ ≡ { ( S , A , P , rw = φ ·w , d0 , γ ) |w ∈ Rn } , ( 3 ) and given SFs ψπ ( s , a ) of a policy π , an efficient form of GPE on the task rw for policy π can be performed as follows : ψπ ( s , a ) > ·w = Qπrw ( s , a ) , ( 4 ) where Qπrw ( s , a ) is the state-action value function of π on the task rw . And , after performing GPE for all the policies π in a finite policy set Π , following Barreto et al . ( 2017 ) , an efficient form of GPI can be performed as follows : πGPIΠ ( s ) ∈ arg max a∈A max π∈Π Qπrw ( s , a ) . ( 5 ) We will refer to this specific use of SFs and value-based action selection for performing GPE & GPI as simply GPE & GPI throughout the rest of this study . Note that πGPI will in general outperform all the policies in Π , and that the actions selected by πGPI on a state may not coincide with any of the actions selected by π ∈ Π on that state . Hence , the policy space that can be attained by GPI can in principle be a lot larger than , e.g. , the space accessible by calling policies sequentially from the original set . 3 PROBLEM FORMULATION AND THEORETICAL ANALYSIS . GPE & GPI provide a guarantee that , for any reward function linear in the features , πGPI is at least as good as any of the policies π from the “ base set ” which was used to construct it . While this is an appealing guarantee of monotonic improvement , it does not say much , for two reasons . First , it is not clear how big an improvement can be expected for different tasks . More importantly , it leaves open the question of how one should choose base policies in order to ensure as much improvement as possible . After all , if we had a weak set of policies and we simply matched their value with πGPI , this may not be very useful . We will now show that , under certain assumptions , having access to a specific set of diverse policies , which we call a set of independent policies , can allow for instantaneously achieving high-level performance on all possible downstream tasks . Let us start by assuming that we are interested in a set of MDPsMφ , as defined in ( 3 ) , with deterministic transition functions ( the reason for the determinism assumption will become clear by the end of this section ) . For convenience , we also restrict the possible w values from Rn toW , where W is the surface of the ` 2 n-dimensional ball . Note that this choice does not alter the optimal policies of the MDPs inMφ , as an optimal policy is invariant with respect to the scale of the reward and W contains all possible directions . Next , we assume that the features φi that make up the feature vectors φ form a set of independent features ( SIF ) , defined as follows : Definition 1 ( SIF ) . A set of features Φ = { φi|φi : S × A × S → { 0 , C } , C ∈ R+ } ni=1 is called independent if , for any feature φi ∈ Φ and any initial state s0 ∼ d0 , we have : ( i ) φi ( s0 , a0 , s1 ) = 0 ∀a0 ∈ A and ∀s1 ∼ P ( s0 , a0 , · ) , and ( ii ) there exists at least one trajectory , starting from s0 , in which all the states associated with φi ( st , at , st+1 ) = C are visited , while the states associated with φj ( st , at , st+1 ) = C , ∀j 6= i , are not visited . It should be noted that a specific instantiation of this definition is the case where each feature is set to a positive constant at certain independently reachable state/states and zero elsewhere , which is the most common instantiation of feature vectors used in previous related work ( Barreto et al. , 2017 ; 2020 ) . We define the performance of an arbitrary policy π on a task rw as : Jπrw ≡ Eπ , P [ ∞∑ t=0 rw ( St , At , St+1 ) ∣∣∣S0 ∼ d0 ] , ( 6 ) whereEπ , P [ · ] denotes the expectation over trajectories induced by π and P . Note that Jπrw is a scalar corresponding to the expected undiscounted return of policy π under the initial state distribution d0 , which is the expected total reward obtained by π when starting from s0 ∼ d0 . We are now ready to formalize the problem we want to tackle : Problem Formulation . Given a set of MDPsMφ with deterministic transition functions and a SIF , we want to construct a set of n policies Πn = { πi } ni=1 , such that the performance of the policy πGPIΠn obtained by performing GPI on that set will maximize ( 6 ) for all rewards rw , where w ∈ W . That is , we want to solve the following problem : arg max Πn⊆Π J πGPIΠn rw for all w ∈ W. ( 7 ) It should be noted that the performance measure provided in ( 6 ) only measures the expected total reward and thus can not capture the optimality of the GPI policy . For instance , this measure can not distinguish between two policies that achieve the same expected total reward in a different number of time steps . However , Theorem 2 in Barreto et al . ( 2017 ) implies that , in general , the only way to guarantee the optimality of the GPI policy is to construct a behavior basis that contains all possible policies induced by all w ∈ W . Since there are infinitely many w values , this is impractical . Thus , in this study , we only consider GPI policies that maximize the expected total reward ( 6 ) . As a solution candidate to the problem in ( 7 ) , we now focus on a specific set of deterministic policies , called set of independent policies ( SIP ) , that are able to obtain features independently of each other : Definition 2 ( SIP ) . Let Φ = { φi } ni=1 be a SIF and let Π = { πi } ni=1 be a set of deterministic policies that are induced by each of the features in Φ. Π is defined to be a SIP if its elements , πi , satisfy : φj ( st , at , st+1 ) = φj ( s0 , a0 , s1 ) ∀j 6= i , ∀i , ∀s0 ∼ d0 and ∀t ∈ { 1 , . . . , T } , ( 8 ) where T is the horizon in episodic environments and T → ∞ in non-episodic ones , a0 = πi ( s0 ) and ( st , at , st+1 ) Tt=1 is the sequence of state-action-state triples generated by πi ’ s interaction with the environment . In general , having a SIP in a set of MDPs with stochastic transition functions is not possible , as the stochasticity can prevent ( 8 ) from holding . Thus , the assumption of a set of MDPs with deterministic transition functions is critical to our analysis . An immediate consequence of having this set of policies is that the corresponding SFs can be expressed in a simpler form , as follows : Lemma 1 . Let Φ be a SIF and let πi be a policy that is induced by the feature φi ∈ Φ and is a member of a SIP Π . Then , the entries of the SF ψπi of policy πi has the following form : ψπij ( s , a ) = { ψπii ( s , a ) , if i = j 0 , otherwise . ( 9 ) Due to the space constraints , we provide all the proofs in the supp . material . Lemma 1 implies that once we have a SIP , the SFs take the much simpler form ( 9 ) , which can allow for the GPI policy to maximize performance on all possible tasks according to the measure in ( 6 ) , solving the optimization objective in ( 7 ) : Theorem 1 . Let Φ be a SIF and let Π be a SIP induced by each of the features in Φ . Then , the GPI policy πGPIΠ is a solution to the optimization problem defined in ( 7 ) . Theorem 1 implies that having access to a SIP which consists of only n policies , where n is the dimensionality of φ , and applying the policy composition operator GPI on top , allows for instantaneously achieving maximum performance across all possible downstream tasks . Considering the fact that there are infinitely many such tasks , this provides a significant gain in the number of policies that are required to be learned for full downstream task coverage .

Summary: The paper focuses on reinforcement learning problems with known successor features and rewards expressible as their linear combination. Building on recent research, it presents a concept of independent features and independent policies and way to construct them. Theoretically it shows that the set of independent policies and their combination with GPE & GPI is enough to solve any induced task. Experimentally, the authors verify the theory and compare to existing approaches to create policy sets, outperforming all. They also provide a set of relevant questions and answers, supported by separate experiments. Finally, they perform experiments in problems without the linear combination assumption and lifelong RL setting, with positive results.