Adapt-to-Learn: Policy Transfer in Reinforcement Learning

1 INTRODUCTION . Lack of principled mechanisms to quickly and efficiently transfer policies learned between domains has become the major bottleneck in Reinforcement Learning ( RL ) . This inability to transfer or adapt policies is one major reason why RL has still not proliferated physical application like robotics . Since RL agents can not quickly transfer policies , the agent is forced to learn every task from scratch , which is both time and sample expensive . Warm-start , a method in which weights from one neural network are transferred to another , has been reasonably successful for supervised learning . However , this method can often lead to mixed and even negative results in RL ( Joshi & Chowdhary , 2018 ; Taylor & Stone , 2009 ) . Our main contribution is an algorithm to transfer policies between tasks with significant differences in state transitions via a policy adaptation mechanism . Unlike the majority of existing work in transfer learning for RL , our approach does not merely use the transferred policy to warm start ( initialize the parameter of the target network with learned source network ) policy learning in the target domain . Neither does it rely on a multitude of simulations across randomly generated source domains . Instead , we combine supervised reference trajectory tracking and unsupervised reinforcement learning to adapt the source policy to the target domain directly . We show through theory and experiments that our method enjoys significantly reduced sample complexity in solving the task . Adapt-to-Learn is inspired by the fact that combined adaptation of behaviors and learning through experience is a primary mechanism of learning in biological creatures ( Krakauer & Mazzoni , 2011 ; Fryling et al. , 2011 ) .Inspired by this ability of biological creatures , we seek to answer the question : Will learning to combine intrinsic adaptation reward with environment reward lead to more efficient transfer of policies between domains ? Imitation Learning ( IL ) ( Duan et al. , 2017 ; Zhu et al. , 2018 ) seems to play a crucial part in biological learning , and as such has been widely studied in RL . However , the key is , when presented with a new situation , animals do not just imitate , but quickly adapt existing behaviors , and improve them through further experience . In particular , an animal learning to walk on a different terrain does not just imitate its existing gait , but adapts it to the new environment.The theory behind such adaptation in reference tracking control problems has been typically restricted to deterministic dynamical systems with well-defined reference trajectories ( Åström & Wittenmark , 2013 ; Chowdhary et al. , 2013 ) . This ability to adapt and incorporate further learning through optimization on the environment reward is one key difference between our method and ex- isting imitation learning and Guided Policy Search ( GPS ) methods ( Levine & Koltun , 2013 ) . Unlike IL and GPS , our method transfers policies between task with significant differences in the transition models . Moreover , by mixing environment reward with intrinsic adaptation rewards , we ensure that the agent quickly adapts and also learns to acquire skills beyond what the source policy can teach . We posit that the presented method can be the foundation of a broader class of RL algorithms that can choose seamlessly between learning through RL to supervised adaptive imitation . Our empirical results show that approach is capable of robustly transferring policies between tasks , even in the presence of nonlinear and time-varying differences in the dynamic model of the systems . In particular , we show that it suffices to execute adapted greedy policies to ensure −optimal behavior in the target domain . Related work : D-RL has recently enabled agents to learn policies for complex robotic tasks in simulation ( Peng et al. , 2016 ; 2017b ; Liu & Hodgins , 2017 ; Heess et al. , 2017 ) . However , D-RL has been plagued by the curse of sample complexity . Therefore , the capabilities demonstrated in the simulated environment are hard to replicate in the real world . This learning inefficiency of RL has led to significant work in the field of TL ( Taylor & Stone , 2009 ) . A significant body of literature on transfer in RL is focused on initialized RL in the target domain using learned source policy ; known as jump-start/warm-start methods ( Taylor et al. , 2005 ; Ammar et al. , 2012 ; 2015 ) . Some examples of these transfer architectures include transfer between similar tasks ( Banerjee & Stone , 2007 ) , transfer from human demonstrations ( Peters & Schaal , 2006 ) and transfer from simulation to real ( Peng et al. , 2017a ; Ross et al. , 2011 ; Yan et al. , 2017 ) . Efforts have also been made in exploring accelerated learning directly on real robots , through Guided Policy Search ( GPS ) ( Levine et al. , 2015 ) and parallelizing the training across multiple agents using meta-learning ( Levine et al. , 2016 ; Nagabandi et al. , 2018 ; Zhu et al. , 2018 ) . Sim-to-Real transfers have been widely adopted in the recent works and can be viewed as a subset of same domain transfer problems . Daftry et al . ( Daftry et al. , 2016 ) demonstrated the policy transfer for control of aerial vehicles across different vehicle models and environments . Christiano et al . ( Christiano et al. , 2016 ) transferred policies from simulation to real using an inverse dynamics model estimated interacting with the real robot . Through learning over an adversarial loss , the agents are trained to achieve robust policies across various environments ( Wulfmeier et al. , 2017 ) . However , these and other reported architectures do not necessarily lead to improved sample efficiency , handle relatively minor changes in the transition model , and are even known to cause negative transfer . In contrast , our approach directly adapts the source policies to target with significant transition model difference while interacting with the environment . It enjoys empirically and rigorously proven sample efficiency guarantees of order O ( nH ) , depending polynomially on the horizon length “ H ” . 2 PRELIMINARIES . Consider a finite horizon MDP defined as a tupleM = ( S , A , P , R , ρ0 , γ ) , where S denote set of continuous states ; A is a set of continuous bounded actions , P : S ×A×S → R+ is state transition probability distribution of reaching s′ upon taking action a in s , ρ0 : S → R+ is the distribution over initial states s0 and R : S × A → R+ is deterministic reward function and H be the finite horizon of the problem . Let π ( a|s ) : S ×A → [ 0 , 1 ] be stochastic policy over continuous state and action space . The action from policy is a draw from this distribution ai ∼ π ( ai|si ) . The agent ’ s goal is to find a policy π ? which maximize the total return . The total return starting from states s0 ∼ ρ0 under a policy π is ηπ ( s0 ) = Es0 , a0 , ... ( H∑ t=0 r ( st , at ) ) ( 1 ) where , s0 ∼ ρ0 , at ∼ π ( at|st ) and st+1 ∼ p ( st+1|st , at ) . We formalize the underlying problem of policy transfer by considering a source and target MDP as follows , MS = ( S , A , P , R , ρ0 , γ ) S , MT = ( S , A , P , R , ρ0 , γ ) T , each with its own state , action space and transition model respectively . We will mainly focus on the problem of same domain transfer in this paper , where the state and action space are analogous S ( S ) = S ( T ) = S ∈ Rm and A ( T ) = A ( S ) = A ∈ Rk , but the source and target state transition models differ significantly due to unmodeled dynamics or external environment interactions . Furthermore , an example of how the method can extend to cross-domain transfer using manifold alignment ( Joshi & Chowdhary , 2018 ; Wang & Mahadevan , 2009 ) is in the supplementary . Let π∗ be a parameterized optimal stochastic policy for source MDP MS . The policy π∗ can be obtained using any available RL methods ( Sutton et al. , 2000 ; Schulman et al. , 2015 ; 2017 ) . In this work , we have used the Proximal Policy Optimization ( PPO ) ( Schulman et al. , 2017 ) algorithm to generate the source optimal policy . We will use the following definition of the state value function V π defined under any policy π V π ( st ) = Eat , st+1 , at+1 , ... ( H∑ i=0 r ( si+t , ai+t ) ) The associated optimal value function fo source MDPMS is V π∗ for all s ∈ S. Let T : Rn → Rn be the Bellman update operator defined as ( T f ) ( s ) = max a∈A [ r ( s , a ) + γ E s′∼P ( s , a ) f ( s′ ) ] ( 2 ) Optimal value function V π ∗ satisfies the Bellman Equation- ( 2 ) such that V π ∗ ( s ) = T V π∗ ( s ) , ∀ s . 3 ALGORITHMIC CONTRIBUTION : TRANSFER THROUGH ADAPTATION . 3.1 ADAPTATION AS A MECHANISM FOR POLICY TRANSFER IN RL . In this paper , we approach the problem of transfer learning for RL through adaptation of previously learned policies to related tasks . Our approach to adaptation is by enabling the agent to learn the best mixture of imitation and learning from environment reward.Our method differs from RL transfer methods that rely on jump-starts or direct imitation ( Zhu et al. , 2018 ; Duan et al. , 2017 ) in a key way : We do not aim to emulate the source optimal policy itself in the target domain , but the source transitions under optimal policy . To make this point , we demonstrate in our empirical results that the source policy , when used directly , does not produce sensible behaviors of the agent in the target domain when transition models are significantly different.On the other hand , our method uses the source transitions projected onto the target task as reference exploration trajectories , helping to adapt and optimize the source policy to the target domain efficiently . In canonical reinforcement learning , the goal is to learn the optimal policy π∗ which maximizes the future cumulative reward when the reward function and the transition function of the MDP are unknown . In statistical RL , this leads to the ( in ) famous explore-exploit tradeoff.Yet , this learnthrough-experience approach has demonstrated significant potential to enable general solutions to control of complex physical systems for which first principles-based models are not easily obtainable . However , one of the key challenges in RL has been the high sample complexity of obtaining a reasonable policy . In other words , RL , and specifically Deep RL , require an absurdly huge amount of interactions with the environment ( experience ) to find reasonable policies . This high learning inefficiency is in apparent contrast with the efficient learning demonstrated by humans and many biological creatures . For examples , octopuses are known to learn to solve complex problems such as opening jars and squeezing through puzzles with just a few trials . This ability to quickly transfer and adapt learned policies between related tasks is likely a key to general intelligence that seems to be missing from RL . Our goal in the rest of this paper is to show that an algorithm that can judiciously combine adaptation to changes and learning new skills is capable of avoiding brute force random exploration to a large extent and be significantly less sample expensive .

Summary: This paper tackles the problem of transferring a policy from source to target MDP, which differ in the state transition function. The idea is to add an additional cost that is the KL divergence between the trajectory likelihood under target policy (being learned) and target dynamics and the trajectory likelihood under the source policy (assumed optimal and deterministic) and source dynamics. The intuition is that the target policy will learn to match the state distribution of the optimal source policy. Results on MuJoCo locomotion robots with varying physics show that the proposed method performs better on target than warm-started RL or learning from scratch.