Universal Successor Features Approximators

1 INTRODUCTION . Reinforcement learning ( RL ) provides a general framework to model sequential decision-making problems with sparse evaluative feedback in the form of rewards . The recent successes in deep RL have prompted interest in increasingly more complex tasks and a shift in focus towards scenarios in which a single agent must solve multiple related problems , either simultaneously or sequentially . This paradigm is formally known as multitask RL ( Taylor and Stone , 2009 ; Teh et al. , 2017 ) . One of the benefits of learning about multiple tasks at the same time is the possibility of transferring knowledge across tasks ; in essence , this means that by jointly learning about a set of tasks one should be able to exploit their common structure to speed up learning and induce better generalisation ( Taylor and Stone , 2009 ; Lazaric , 2012 ) . A particularly interesting instance of transfer is the generalisation to new , unseen tasks . This potentially allows an agent to perform a task with little or no learning by leveraging knowledge from previously learned tasks . In this paper we will be exploring this scenario . Consider an RL agent in a persistent environment trying to master a number of tasks . In order to generalise to unseen tasks , the agent needs to be able to identify and exploit some common structure underlying the tasks . Two possible sources of structure in this scenario are : i ) some similarity between the solutions of the tasks , either in the policy or in the associated value-function space , and ii ) the shared dynamics of the environment ( e.g. , physics ) . In this paper we will attempt to build an agent that can make use of both types of structure . For this , we will build on two frameworks that exploit these structures in isolation . The first one are Schaul et al. ’ s ( 2015 ) universal value function approximators ( UVFAs ) . UVFAs extend the notion of value functions to also include the description of a task , thus directly exploiting the common structure in the associated optimal value functions . The second framework we build upon exploits the common structure in the environment and capitalises on the power of dynamic programming . Barreto et al. ’ s ( 2017 ) framework is based on two core ∗Corresponding author : borsa @ google.com concepts : successor features ( SFs ) , a representation scheme that allows a policy to be evaluated on any task of a given format , and generalised policy improvement ( GPI ) , a generalisation of dynamic programming ’ s classic operator that uses a set of policies instead of a single one . UVFAs and SF & GPI generalise to new tasks in quite different , and potentially complementary , ways . UVFAs aim to generalise across the space of tasks by exploiting structure in the underlying space of value functions . In contrast , SF & GPI strategy is to exploit the structure of the RL problem itself . In this paper we propose a model that exhibits both types of generalisation . The basic insight is to note that SFs are multi-dimensional value functions , so we can extend them in the same way a universal value function extends their unidimensional counterparts . We call the resulting model universal successor features approximators , or USFAs for short . USFA is a strict generalisation of its precursors . Specifically , we show that by combining USFAs and GPI we can recover both UVFAs and SF & GPI as particular cases . This opens up a new spectrum of possible approaches in between these two extreme cases . We discuss the challenges involved in training a USFA and demonstrate the practical benefits of doing so on a large-scale domain in which the agent has to navigate in a three-dimensional environment using only images as observations . 2 BACKGROUND . In this section we present some background material , formalise the scenario we are interested in , and briefly describe the methods we build upon . 2.1 MULTITASK REINFORCEMENT LEARNING . We consider the usual RL framework : an agent interacts with an environment and selects actions in order to maximise the expected amount of reward received in the long run ( Sutton and Barto , 1998 ) . As usual , we assume that such an interaction can be modeled as a Markov decision process ( MDP , Puterman , 1994 ) . An MDP is defined as a tuple M ≡ ( S , A , p , R , γ ) where S andA are the state and action spaces , p ( ·|s , a ) gives the next-state distribution upon taking action a in state s , R ( s , a , s′ ) is a random variable representing the reward received at transition s a−→ s′ , and γ ∈ [ 0 , 1 ) is a discount factor that gives smaller weights to future rewards . As mentioned in the introduction , in this paper we are interested in the multitask RL scenario , where the agent has to solve multiple tasks . Each task is defined by a reward function Rw ; thus , instead of a single MDP M , our environment is a set of MDPs that share the same structure except for the reward function . Following Barreto et al . ( 2017 ) , we assume that the expected one-step reward associated with transition s a−→ s′ is given by E [ Rw ( s , a , s ′ ) ] = rw ( s , a , s ′ ) = φ ( s , a , s′ ) > w , ( 1 ) where φ ( s , a , s′ ) ∈ Rd are features of ( s , a , s′ ) and w ∈ Rd are weights . The features φ ( s , a , s′ ) can be thought of as salient events that may be desirable or undesirable to the agent , such as for example picking up an object , going through a door , or knocking into something . In this paper we assume that the agent is able to recognise such events—that is , φ is observable— , but the solution we propose can be easily extended to the case where φ must be learned ( Barreto et al. , 2017 ) . Our solution also applies to the case where ( 1 ) is only approximately satisfied , as discussed by Barreto et al . ( 2018 ) . Given w ∈ Rd representing a task , the goal of the agent is to find a policy πw : S 7→ A that maximises the expected discounted sum of rewards , also called the return G ( t ) w = ∑∞ i=0 γ iR ( t+i ) w , where R ( t ) w = Rw ( St , At , St+1 ) is the reward received at the tth time step . A principled way to address this problem is to use methods derived from dynamic programming ( DP ) , which heavily rely on the concept of a value function ( Puterman , 1994 ) . The action-value function of a policy π on task w is defined as Qπw ( s , a ) ≡ Eπ [ G ( t ) w |St = s , At = a ] , where Eπ [ · ] denotes expected value when following policy π . Based on Qπw we can compute a greedy policy π ′ ( s ) ∈ argmaxaQπw ( s , a ) ; one of the fundamental results in DP guarantees that Qπ ′ w ( s , a ) ≥ Qπw ( s , a ) for all ( s , a ) ∈ S ×A . The computation of Qπw ( s , a ) and π ′ are called policy evaluation and policy improvement ; under certain conditions their successive application leads to the optimal value function Q∗w , from which one can derive an optimal policy for task w as π∗w ( s ) ∈ argmaxaQ∗w ( s , a ) ( Sutton and Barto , 1998 ) . As a convention , in this paper we will add a tilde to a symbol to indicate that the associated quantity is an approximation ; we will then refer to the respective tunable parameters as θ . For example , the agent computes an approximation Q̃πw ≈ Qπw by tuning θQ . 2.2 TRANSFER LEARNING . Here we focus on one aspect of multitask RL : how to transfer knowledge to unseen tasks ( Taylor and Stone , 2009 ; Lazaric , 2012 ) . Specifically , we ask the following question : how can an agent leverage knowledge accumulated on a set of tasksM⊂ Rd to speed up the solution of a new task w′ /∈M ? In order to investigate the question above we recast it using the formalism commonly adopted in learning . Specifically , we define a distribution Dw over Rd and assume the goal is for the agent to perform as well as possible under this distribution . As usual , we assume a fixed budget of sample transitions and define a training setM ∼ Dw that is used by the agent to learn about the tasks of interest . We also define a test setM′ ∼ Dw and use it to assess the agent ’ s generalisation—that is , how well it performs on the distribution of MDPs induced by Dw . A natural way to address the learning problem above is to use Schaul et al. ’ s ( 2015 ) universal valuefunction approximators ( UVFAs ) . The basic insight behind UVFAs is to note that the concept of optimal value function can be extended to include as one of its arguments a description of the task ; an obvious way to do so in the current context is to define the function Q∗ ( s , a , w ) : S ×A× Rd 7→ R as the optimal value function associated with task w. The function Q∗ ( s , a , w ) is called a universal value function ( UVF ) ; a UVFA is then the corresponding approximation , Q̃ ( s , a , w ) . When we define transfer as above it becomes clear that in principle a sufficiently expressive UVFA can identify and exploit structure across the joint space S ×A×Rd . In other words , a properly trained UVFA should be able to generalise across the space of tasks . A different way of generalising across tasks is to use Barreto et al. ’ s ( 2017 ) framework , which builds on assumption ( 1 ) and two core concepts : successor features ( SFs ) and generalised policy improvement ( GPI ) . The SFs ψ ∈ Rd of a state-action pair ( s , a ) under policy π are given by ψπ ( s , a ) ≡ Eπ [ ∞∑ i=t γi−tφi+1 |St = s , At = a ] . ( 2 ) SFs allow one to immediately compute the value of a policy π on any task w : it is easy to show that , when ( 1 ) holds , Qπw ( s , a ) = ψ π ( s , a ) > w . It is also easy to see that SFs satisfy a Bellman equation in which φ play the role of rewards , so ψ can be learned using any RL method ( Szepesvári , 2010 ) . GPI is a generalisation of the policy improvement step described in Section 2.1 . The difference is that in GPI the improved policy is computed based on a set of value functions rather than on a single one . Suppose that the agent has learned the SFs ψπi of policies π1 , π2 , ... , πn . When exposed to a new task defined by w , the agent can immediately compute Qπiw ( s , a ) = ψ πi ( s , a ) > w . Let the GPI policy be defined as π ( s ) ∈ argmaxaQmax ( s , a ) , where Qmax = maxiQπi . The GPI theorem states that Qπ ( s , a ) ≥ Qmax ( s , a ) for all ( s , a ) ∈ S × A . The result also extends to the scenario where we replace Qπi with approximations Q̃πi ( Barreto et al. , 2017 ) .

Summary: The goal here is multi-task learning and generalization, assuming that the expected one-step reward for any member of the task family can be written as $\phi(s,a,s')^T w$. The authors propose universal successor features (USF) $\psi$s, such that the action-value functions Q can be written as $Q(s,a,w,z)=\psi(s,a,z)^T w$, generalizing over mutiple tasks each denoted by $w$, and multiple policies each denoted by $z$. Here, $z$ represents the optimal policy induced by a reward specified by $z$ (from the same set as $w$). Using USFs $\psi$-s, the Q values can be interpolated across policies and tasks. Due to the disentangling of reward and policy generalizations, the training sets for $w$ and $z$ can be independently sampled. The authors further generalize a temporal difference error in these USFs $\psi$s, using the TD error to learn to approximate the $\psi$s by a network (USF Approximator i.e USFA). They then test the generalization capabilities of these USFAs on families of a simple task and a DeepMind Lab based task.