Policy Transfer with Strategy Optimization

1 INTRODUCTION . Recent developments in Deep Reinforcement Learning ( DRL ) have shown the potential to learn complex robotic controllers in an automatic way with minimal human intervention . However , due to the high sample complexity of DRL algorithms , directly training control policies on the hardware still remains largely impractical for agile tasks such as locomotion . A promising direction to address this issue is to use the idea of transfer learning which learns a model in a source environment and transfers it to a target environment of interest . In the context of learning robotic control policies , we can consider the real world the target environment and the computer simulation the source environment . Learning in simulated environment provides a safe and efficient way to explore large variety of different situations that a real robot might encounter . However , due to the model discrepancy between physics simulation and the real-world environment , also known as the Reality Gap ( Boeing & Bräunl , 2012 ; Koos et al. , 2010 ) , the trained policy usually fails in the target environment . Efforts have been made to analyze the cause of the Reality Gap ( Neunert et al. , 2017 ) and to develop more accurate computer simulation ( Tan et al. , 2018 ) to improve the ability of a policy when transferred it to real hardware . Orthogonal to improving the fidelity of the physics simulation , researchers have also attempted to cross the reality gap by training more capable policies that succeed in a large variety of simulated environments . Our method falls into the second category . To develop a policy capable of performing in various environments with different governing dynamics , one can consider to train a robust policy or to train an adaptive policy . In both cases , the policy is trained in environments with randomized dynamics . A robust policy is trained under a range of dynamics without identifying the specific dynamic parameters . Such a policy can only perform well if the simulation is a good approximation of the real world dynamics . In addition , for more agile motor skills , robust policies may appear over-conservative due to the uncertainty in the training environments . On the other hand , when an adaptive policy is used , it learns to first identify , implicitly or explicitly , the dynamics of its environment , and then selects the best action according to the identified dynamics . Being able to act differently according to the dynamics allows the adaptive policy to achieve higher performance on a larger range of dynamic systems . However , when the target dynamics is notably different from the training dynamics , it may still produce sub-optimal results for two reasons . First , when a sequence of novel observations is presented , the learned identification model in an adaptive policy may produce inaccurate estimations . Second , even when the identification model is perfect , the corresponding action may not be optimal for the new situation . In this work , we introduce a new method that enjoys the versatility of an adaptive policy , while avoiding the challenges of system identification . Instead of relating the observations in the target environment to the similar experiences in the training environment , our method searches for the best policy directly based on the task performance in the target environment . Our algorithm can be divided to two stages . The first stage trains a family of policies , each optimized for a particular vector of dynamic parameters . The family of policies can be parameterized by the dynamic parameters in a continuous representation . Each member of the family , referred to as a strategy , is a policy associated with particular dynamic parameters . Using a locomotion controller as an example , a strategy associated with low friction coefficient may exhibit cautious walking motion , while a strategy associated with high friction coefficient may result in more aggressive running motion . In the second stage we perform a search over the strategies in the target environment to find the one that achieves the highest task performance . We evaluate our method on three examples that demonstrate transfer of a policy learned in one simulator DART , to another simulator MuJoCo . Due to the differences in the constraint solvers , these simulators can produce notably different simulation results . A more detailed description of the differences between DART and MuJoCo is provided in Appendix A . We also add latency to the MuJoCo environment to mimic a real world scenario , which further increases the difficulty of the transfer . In addition , we use a quadruped robot simulated in Bullet to demonstrate that our method can overcome actuator modeling errors . Latency and actuator modeling have been found to be important for Sim-to-Real transfer of locomotion policies ( Tan et al. , 2018 ; Neunert et al. , 2017 ) . Finally , we transfer a policy learned for a robot composed of rigid bodies to a robot whose end-effector is deformable , demonstrating the possiblity of using our method to transfer to problems that are challenging to model faithfully . 2 RELATED WORK . While DRL has demonstrated its ability to learn control policies for complex and dynamic motor skills in simulation ( Schulman et al. , 2015 ; 2017 ; Peng et al. , 2018a ; 2017 ; Yu et al. , 2018 ; Heess et al. , 2017 ) , very few learning algorithms have successfully transferred these policies to the real world . Researchers have proposed to address this issue by optimizing or learning a simulation model using data from the real-world ( Tan et al. , 2018 ; 2016 ; Deisenroth & Rasmussen , 2011 ; Ha & Yamane , 2015 ; Abbeel & Ng , 2005 ) . The main drawback for these methods is that for highly agile and high dimensional control problems , fitting an accurate dynamic model can be challenging and data inefficient . Complementary to learning an accurate simulation model , a different line of research in sim-to-real transfer is to learn policies that can work under a large variety of simulated environments . One common approach is domain randomization . Training a robust policy with domain randomization has been shown to improve the ability to transfer a policy ( Tan et al. , 2018 ; Tobin et al. , 2017 ; Rajeswaran et al. , 2016 ; Pinto et al. , 2017 ) . Tobin et al . ( 2017 ) trained an object detector with randomized appearance and applied it in a real-world gripping task . Tan et al . ( 2018 ) showed that training a robust policy with randomized dynamic parameters is crucial for transferring quadruped locomotion to the real world . Designing the parameters and range of the domain to be randomized requires specific knowledge for different tasks . If the range is set too high , the policy may learn a conservative strategy or fail to learn the task , while a small range may not provide enough variation for the policy to transfer to real-world . A similar idea is to train an adaptive policy with the current and the past observations as input . Such an adaptive policy is able to identify the dynamic parameters online either implicitly ( OpenAI et al. , 2018 ; Peng et al. , 2018b ) or explicitly ( Yu et al. , 2017 ) and apply actions appropriate for different system dynamics . Recently , adaptive policies have been used for sim-to-real transfer , such as in-hand manipulation tasks ( OpenAI et al. , 2018 ) or non-prehensile manipulation tasks ( Peng et al. , 2018b ) . Instead of training one robust or adaptive policy , Zhang et al . ( 2018 ) trained multiple policies for a set of randomized environments and learned to combine them linearly in a separate set of environments . The main advantage of these methods is that they can be trained entirely in simulation and deployed in real-world without further fine-tuning . However , policies trained in simulation may not generalize well when the discrepancy between the target environment and the simulation is too large . Our method also uses dynamic randomization to train policies that exhibit different strategies for different dynamics , however , instead of relying on the simulation to learn an identification model for selecting the strategy , we propose to directly optimize the strategy in the target environment . A few recent works have also proposed the idea of training policies in a source environment and fine-tune it in the target environment . For example , Cully et al . ( 2015 ) proposed MAP-Elite to learn a large set of controllers and applied Bayesian optimization for fast adaptation to hardware damages . Their approach searches for individual controllers for discrete points in a behavior space , instead of a parameterized family of policies as in our case , making it potentially challenging to be applied to higher dimensional behavior spaces . Rusu et al . ( 2016 ) used progressive network to adapt the policy to new environments by designing a policy architecture that can effectively utilize previously learned representations . Chen et al . ( 2018 ) learned an implicit representation of the environment variations by optimizing a latent policy input for each discrete instance of the environment . They showed that fine-tuning on this learned policy achieved improved learning efficiency . In contrast to prior work in which the fine-tuning phase adjusts the neural network weights in the target environment , we optimize only the dynamics parameters input to the policy . This allows our policies to adapt to the target environments with less data and to use sparse reward signal . 3 BACKGROUND . We formulate the motor skill learning problem as a Markov Decision Process ( MDP ) , M = ( S , A , r , P , p0 , γ ) , where S is the state space , A is the action space , r : S × A 7→ R is the reward function , P : S × A 7→ S is the transition function , p0 is the initial state distribution and γ is the discount factor . The goal of reinforcement learning is to find a control policy π : S 7→ A that maximizes the expected accumulated reward : JM ( π ) = Eτ= ( s0 , a0 , ... , sT ) ∑T t=0 γ tr ( st , at ) , where s0 ∼ p0 , at ∼ π ( st ) and st+1 = P ( st , at ) . In practice , we usually only have access to an observation of the robot that contains a partial information of the robot ’ s state . In this case , we will have a Partially-Observable Markov Decision Process ( POMDP ) and the policy would become π : O 7→ A , where O is the observation space . In the context of transfer learning , we can define a source MDPMs and a target MDPMt and the goal would be to learn a policy πs forMs such that it also works well onMt . In this work , P is regarded as a parameterized space of transition functions , st+1 = Pµ ( st , at ) , where µ is a vector of physical parameters defining the dynamic model ( e.g . friction coefficient ) . The transfer learning in this context learns a policy under Ps and transfers to Pt , where Ps 6= Pt . 4 METHODS . We propose a new method for transferring a policy learned in simulated environment to a target environment with unknown dynamics . Our algorithm consists of two stages : learning a family of policies and optimizing strategy .

Summary: This paper presents a novel approach for adapting a policy learned with domain randomization to the target domain. The parameters for domain randomization are explicitly used as input to the network learning the policy. When run in the target domain, CMA-ES is used to search over these domain parameters to find the ones that lead to the policy with the best returns in the target domain.