Universal Successor Features for Transfer Reinforcement Learning

1 INTRODUCTION . Deep Reinforcement Learning ( RL ) has been able to achieve human-level performance in many domains , such as playing Atari games ( Mnih et al. , 2015 ) and controlling robotic arms ( Levine et al. , 2016 ) . However , these methods often spend a huge amount of time and resources to train a deep model to perform a single task . Currently applying knowledge gained from performing a single task to performing another related task remains a challenging problem . This is the problem Transfer learning is trying to solve ( Taylor & Stone , 2009 ) . Evidently though transferring knowledge from one task to another would not be possible if the two tasks were completely unrelated . Therefore , in this work , we focus on one particular transfer scenario , where dynamics among tasks remain the same while their goals are different . We elaborate on this in Sec.2 . What can we use as knowledge for transfer in RL ? One form of model-free knowledge is the goalspecific state-values as in general value functions ( Sutton et al. , 2011 ) . However , the state-values ’ dependency on the goal and its corresponding reward structure may be an obstacle when attempting to use these values for transfer . An alternative form of model-based knowledge is to learn the transition dynamics1 , which are independent of the reward function . However , when performing a rollout with the estimated transition dynamics prediction error will accumulate over time , which may limit the utility of this approach in practice . A third form of knowledge that may be suitable for transfer is successor features ( SFs ) ( Dayan , 1993 ; Barreto et al. , 2017 ; 2018 ) , which can recover state-values without explicitly performing a rollout . Briefly speaking , SFs provide an action-contingent summary of the future . This third approach , can not directly generalize to unseen new goals . In this paper , we propose Universal Successor Features ( USFs ) . Similar to how a universal value function ( Schaul et al. , 2015 ) learn to generalize values over goals , USFs learn to generalize successor features over goals . Since SFs for each goal capture certain knowledge about the underlying dynamics of the environment , USFs could learn the shared dynamics and exploit their generality over the goals . Furthermore , USFs are also able to recover values without explicitly performing a rollout . USFs result from decomposing universal state-values into USFs and reward features . Note that while we focus on state-action-values ( or simply action-values ) in this paper in order to tackle 1the action-contingent probability of transitioning from one state to another control problems , this idea can apply to state-values as well . Unlike in Schaul et al . ( 2015 ) , we learn these two decomposed factors jointly by exploring the environment while bootstrapping . Our contributions are three-fold . First , we propose USFs and show how they can be applied to any RL algorithm that learns state-values using a temporal difference method . Second , we propose a flexible end-to-end architecture that is suitable to learn USFs by interacting with the environment . Third , our experiments demonstrate that USFs can greatly accelerate training when learning multiple tasks and can effectively transfer knowledge to new tasks . This paper is organized as follows . In Sec.2 , we introduce USFs together with our end-to-end architecture , training procedure , and we further explain how USFs can be used for transfer Reinforcement Learning . Then , in Sec.3 , we demonstrate the results of applying USFs to a simple gridworld environment and two MoJuCo environments . Afterward , in Sec.4 , we discuss some of the most relevant papers in the literature . Finally we conclusions in Sec.5 . 2 UNIVERSAL SUCCESSOR FEATURES . Consider a Markov decision process ( MDP ) with state space S , action spaceA , and transition function p where p ( s′|s , a ) is the probability2 of transitioning to s′ ∈ S when action a ∈ A is taken in state s ∈ S . We borrow the extension of the MDP framework to goals and state-dependent discounting from Schaul et al . ( 2015 ) . For each goal g ∈ G ( often G ⊆ S but this is not an explicit requirement of this framework ) , define the pseudo-reward function rg ( s , a , s′ ) and pseudo-discount function γg ( s ) ∈ [ 0 , 1 ] . To define an episodic problem in this framework we can define γg ( s ) to be 0 if s is a terminal state with respect to g. For any policy π : S 7→ A and fixed goal g , the action-value function is defined as Qπg ( s , a ) = Eπ [ ∞∑ t=0 rg ( St , At , St+1 ) t∏ k=1 γg ( Sk ) ∣∣∣∣∣S0 = s , A0 = a ] . As we now show , we can factorize Qπg into successor features and goal-specific features . 2.1 TRANSFER VIA UNIVERSAL SUCCESSOR FEATURES . As with Kulkarni et al . ( 2016 ) and Barreto et al . ( 2017 ) , we assume that the reward function rg ( st , at , st+1 ) can be factorized as rg ( st , at , st+1 ) = φ ( st , at , st+1 ) > wg , ( 1 ) where φ : S × A × S 7→ Rd represents the transitions and wg ∈ Rd are goal-specific features of the reward . Note that in this factorization only wg depends on the goal . In many situations φ ( st , at , st+1 ) can be rewritten as φ ( st+1 ) . To see why , consider a simple gridworld with d cells . Here φ ( st , at , st+1 ) = φ ( st+1 ) can be a one-hot vector indicating the position of the agent at time t+1 , while wg can be a d-dimensional vector whose i-th entry is the immediate reward of reaching the cell i under the goal g. If φ ( st , at , st+1 ) and wg are defined in this way then whatever the previous state st and action at are , as long as the agent ends up in st+1 , the transition is represented by φ ( st+1 ) and the agent will receive the corresponding reward from wg for reaching that state . In other cases where the reward depends on the whole transition , we can easily modify subsequent discussion and model architecture to accommodate this . Using the factorization in Eq . ( 1 ) , the action-value can be rewritten as Qπg ( s , a ) = Eπ [ ∞∑ t=0 φ ( St , At , St+1 ) t∏ k=1 γg ( Sk ) ∣∣∣∣∣S0 = s , A0 = a ] > wg = ψ π g ( s , a ) > wg where ψπg ( s , a ) are the Universal Successor Features ( USFs ) of s and a for goal g. In the above gridworld example , ψπg ( s , a ) would be a d-dimensional vector of discounted expected future state visitations that results after taking action a in state s and thereafter following policy π . 2or probability density in case of continuous state space If wg is given , then the following Bellman equations enable us to learn USFs the same way as we would learn the action-values : Qπg ( s , a ) = Eπ [ rg ( s , a , S′ ) +γg ( S′ ) Qπg ( S′ , A′ ) ] , ψπg ( s , a ) = Eπ [ φ ( s , a , S′ ) +γg ( S′ ) ψπg ( S′ , A′ ) ] . We can approximate ψπ ( s , a ) using a model ψπ ( s , a ; θψ ) parameterized by θψ , so that our method can be used with any algorithm that learns action-values using a temporal difference method3 , including on-policy methods like SARSA ( Sutton , 1996 ) and A2C ( Mnih et al. , 2016 ) as well as off-policy methods like DQN ( Mnih et al. , 2015 ) and DDPG ( Lillicrap et al. , 2016 ) . One remaining problem is that we need to have wg for a given g. One way to accomplish this is to model wg as w ( g ; θw ) and learn the parameters θw . In order to utilize USFs for transfer , we can learn the parameters θ = [ θψ , θw ] from some “ good ” policies and their SFs for a subset of goals , then let the model generalize these learned SFs to the SFs of “ good ” policies of unseen goals . In the next section , we describe how one could train θ . 2.2 TRAINING USFS . Suppose that at time t , we obtain a transition4 { g , st , at , st+1 , rt+1 , γt+1 } where rt+1 = rg ( st , at , st+1 ) and γt+1 = γg ( st , at , st+1 ) . This transition can either come directly from the environment or may be sampled from a replay buffer . Recall that USFs can be applied to any base RL algorithm as long as it involves learning action-values using a temporal difference method . In the case of DQN , which naturally extends to our multi-goal setting ( see Appendix A for more details ) , we update the parameters θQ of the action-value function using ( semi- ) gradient descent on [ Q ( st , at , g ; θQ ) − yt ] 2 where yt = rt+1 + γt+1 max a Q ( st+1 , a , g ; θQ ) ( 2 ) is considered a fixed target action-value ( i.e. , it is treated as a constant rather than a function of θQ ) . Similarly , to learn USFs ’ parameters θ for optimal action-values5 , the next-state action a∗ could be a∗ = argmax a ψ ( st+1 , a , g ; θψ ) > w ( g ; θw ) . Using this , we define the loss on Q to be LQ= [ Q̂t−ψ ( st , at , g ; θψ ) > w ( g ; θw ) ] 2 where Q̂t=rt+1+γt+1ψ ( st+1 , a∗ , g ; θψ ) > w ( g ; θw ) ( 3 ) is the fixed target action-value . Similarly , we define the loss on SF to be Lψ = ∥∥∥ψ̂t −ψ ( st , at , g ; θψ ) ∥∥∥2 2 where ψ̂t = φ ( st , at , st+1 ) + γt+1ψ ( st+1 , a∗ , g ; θψ ) ( 4 ) is considered the fixed target SF . This loss ensures that the learned SF will satisfy its own Bellman equation . As we will show in the experiments , this loss is resilient to the reward structure of the MDP . Although we can separately learn w based on a regression loss from Eq . ( 1 ) , it is less informative in terms of guiding the agent compared to regression based on Eq . ( 3 ) , the actual quantity for decision making ( results in Appendix D ) . Our final loss function is L = LQ + λ · Lψ where λ > 0 is a hyperparameter . With this loss , we can perform updates based on the ( semi- ) gradient∇θL . To see how we can use USFs for transfer , note that because learning is based on the optimal action , Eq . ( 3 ) and ( 4 ) learn the ψ that corresponds to the optimal policy of the given goal . As long as the model is powerful enough , the learned USFs can directly generalize to unseen goals . For each such goal , USFs can either produce a ψ corresponding to an optimal policy for that goal , or provide a reasonable approximation . For the features φ ( st , at , st+1 ) , as discussed above , it is often sufficient to model it as φ ( st+1 ) . When there exists meaningful , natural representations for states , for example , one-hot vectors in a gridworld , we can use them for state representation . If not then we need to learn φ ( s ) . One obvious way to learn φ ( s ) would be to regress r on φ , w , but , as we mentioned above , learning based on r is less informative . Unlike Kulkarni et al . ( 2016 ) who learn the state representations separately with an auto-encoder , we embed the learning of φ into the learning of the USFs , in particular , one hidden layer in the neural networks as shown in Fig.1 . 3This method can also be used with any algorithm that learns state-values to learn successor representations . 4Here we illustrate based on a single transition , but very often we use a mini-batch for training . 5Here we focus on optimal control . In case of on-policy evaluation for a given policy π , the next state action could be a = π ( st+1 ) instead . Algorithm 1 Multi-goal DQN with USFs 1 : Initialize θ and replay buffer D 2 : for each episode do 3 : Get goal g and initial state s0 4 : for each time step t = 0 , · · · , T do 5 : at = Uniform ( |A| ) if Bernoulli ( ) otherwise argmaxaQg ( st , a ; θ ) . -greedy 6 : Execute at and observe next state st+1 and immediate reward rt+1 7 : Store transition { g , st , at , st+1 , rt+1 , γt+1 } in D 8 : Sample random minibatch of transitions { gi , si , ai , si+1 , ri+1 , γi+1 } from D 9 : Compute target Q̂i and ψ̂i according to Eq . ( 3 ) and ( 4 ) . as opposed to yt in Eq . ( 2 ) in case of multi-goal DQN 10 : Perform gradient descent on L w.r.t . θ based on the targets . as opposed to Eq . ( 2 ) 11 : end for 12 : end for Fig.1 shows the neural network structure of our USFs . This version corresponds to the finite action case . After going through part of the USF network , a state s is represented by a d-dimensional vector φ ( s ) ( Kulkarni et al. , 2016 ) . Then it is concatenated with a goal vector to produce a ( |A| × d ) -dimensional SF vector ψ ( s , a , g ) for all |A| actions . On the right-most part of the network , we have a model to compute wg given a goal g. The inner product of ψ and w will be the action-value Q for the corresponding action . As a concrete example , Algo . 1 shows how DQN can be extended to Multi-goal DQN and how USFs can be applied to Multi-goal DQN . There is an easy extension for the continuous action case . The base RL algorithm ( e.g. , DDPG , which also naturally extends to a Multigoal setting ) may have an additional “ actor ” component for the policy . In such cases , our model , with some minor modifications , can accommodate this . We elaborate on this in Appendix B .

Summary: In this paper the authors propose an extension to successor features (SF). Akin to UVFAs, they condition on some goal state by concatenating to the current state after some shared preprocessing. The authors claim three contributions: 1) introducing the USF, 2) proposing an appropriate deep learning architecture for it, and 3) showing experimentally that USFs improve transfer both within a goal set and to novel goals.