Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers

1 INTRODUCTION . Reinforcement learning ( RL ) can automate the acquisition of complex behavioral policies through real-world trial-and-error experimentation . However , many domains where we would like to learn policies are not amenable to such trial-and-error learning , because the errors are too costly : from autonomous driving to flying airplanes to devising medical treatment plans , safety-critical RL problems necessitate some type of transfer learning , where a safer source domain , such as a simulator , is used to train a policy that can then function effectively in a target domain . In this paper , we examine a specific transfer learning scenario that we call domain adaptation , by analogy to domain adaptation problems in computer vision ( Csurka , 2017 ) , where the training process in a source domain can be modified so that the resulting policy is effective in a given target domain . RL algorithms today require a large amount of experience in the target domain . However , for many tasks we may have access to a different but structurally similar source domain . While the source domain has different dynamics than the target domain , experience in the source domain is much cheaper to collect . However , transferring policies from one domain to another is challenging because strategies which are effective in the source domain may not be effective in the target domain . For example , aggressive driving may work well on a dry racetrack but fail catastrophically on an icy road . ∗Equal contribution . While prior work has studied the domain adaptation of observations in RL ( Bousmalis et al. , 2018 ; Ganin et al. , 2016 ; Higgins et al. , 2017 ) , it ignores the domain adaptation of the dynamics . This paper presents a simple approach for domain adaptation in RL , illustrated in Fig . 1 . Our main idea is that the agent ’ s experience in the source domain should look similar to its experience in the target domain . Building off of a probabilistic view of RL , we formally show that we can achieve this goal by compensating for the difference in dynamics by modifying the reward function . This modified reward function is simple to estimate by learning auxiliary classifiers that distinguish sourcedomain transitions from target-domain transitions . Because our method learns a classifier , rather than a dynamics model , we expect it to handle high-dimensional tasks better than model-based methods , a conjecture supported by experiments on the 111-dimensional Ant task . Unlike prior work based on similar intuition ( Koos et al. , 2012 ; Wulfmeier et al. , 2017b ) , a key contribution of our work is a formal guarantee that our method yields a near-optimal policy for the target domain . The main contribution of this work is an algorithm for domain adaptation to dynamics changes in RL , based on the idea of compensating for differences in dynamics by modifying the reward function . We call this algorithm Domain Adaptation with Rewards from Classifiers , or DARC for short . DARC does not estimate transition probabilities , but rather modifies the reward function using a pair of classifiers . We formally analyze the conditions under which our method produces nearoptimal policies for the target domain . On a range of discrete and continuous control tasks , we both illustrate the mechanics of our approach and demonstrate its scalability to higher-dimensional tasks . 2 RELATED WORK . While our work will focus on domain adaptation applied to RL , we start by reviewing more general ideas in domain adaptation , and defer to Kouw & Loog ( 2019 ) for a recent review of the field . Two common approaches to domain adaptation are importance weighting and domain-agnostic features . Importance-weighting methods ( e.g. , ( Zadrozny , 2004 ; Cortes & Mohri , 2014 ; Lipton et al. , 2018 ) ) estimate the likelihood ratio of examples under the target domain versus the source domain , and use this ratio to re-weight examples sampled from the source domain . Similar to prior work on importance weighting ( Bickel et al. , 2007 ; Sønderby et al. , 2016 ; Mohamed & Lakshminarayanan , 2016 ; Uehara et al. , 2016 ) , our method will use a classifier to estimate a probability ratio . Since we will need to estimate the density ratio of conditional distributions ( transition probabilities ) , we will learn two classifiers . Importantly , we will use the logarithm of the density ratio to modify the reward function instead of weighting samples by the density ratio , which is often numerically unstable ( see , e.g. , Schulman et al . ( 2017 , §3 ) ) and led to poor performance in our experiments . Prior methods for applying domain adaptation to RL include approaches based on system identification , domain randomization , and observation adaptation . Perhaps the most established approach , system identification ( Ljung , 1999 ) , uses observed data to tune the parameters of a simulator ( Feldbaum , 1960 ; Werbos , 1989 ; Wittenmark , 1995 ; Ross & Bagnell , 2012 ; Tan et al. , 2016 ; Zhu et al. , 2017b ; Farchy et al. , 2013 ) More recent work has successfully used this strategy to bridge the sim2real gap ( Chebotar et al. , 2019 ; Rajeswaran et al. , 2016 ) . Closely related is work on online system identification and meta-learning , which directly uses the inferred system parameters to update the policy ( Yu et al. , 2017 ; Clavera et al. , 2018 ; Tanaskovic et al. , 2013 ; Sastry & Isidori , 1989 ) . However , these approaches typically require either a model of the environment or a manually-specified distribution over potential test-time dynamics , requirements that our method will lift . Another approach , domain randomization , randomly samples the parameters of the source domain and then finds the best policy for this randomized environment ( Sadeghi & Levine , 2016 ; Tobin et al. , 2017 ; Peng et al. , 2018 ; Cutler et al. , 2014 ) . While often effective , this method is sensitive to the choice of which parameters are randomized , and the distributions from which these simulator parameters are sampled . A third approach , observation adaptation , modifies the observations of the source domain to appear similar to those in the target domain ( Fernando et al. , 2013 ; Hoffman et al. , 2016 ; Wulfmeier et al. , 2017a ) . While this approach has been successfully applied to video games ( Gamrian & Goldberg , 2018 ) and robot manipulation ( Bousmalis et al. , 2018 ) , it ignores the fact that the source and target domains may have differing dynamics . Finally , our work is similar to prior work on transfer learning ( Taylor & Stone , 2009 ) and metalearning in RL , but makes less strict assumptions than most prior work . For example , most work on meta-RL ( Killian et al. , 2017 ; Duan et al. , 2016 ; Mishra et al. , 2017 ; Rakelly et al. , 2019 ) and some work on transfer learning ( Perkins et al. , 1999 ; Tanaka & Yamamura , 2003 ; Sunmola & Wyatt , 2006 ) assume that the agent has access to many source tasks , all drawn from the same distribution as the target task . Selfridge et al . ( 1985 ) ; Madden & Howley ( 2004 ) assume a manually-specified curriculum of tasks , Ravindran & Barto ( 2004 ) assume that the source and target domains have the same dynamics locally , and Sherstov & Stone ( 2005 ) assume that the set of actions that are useful in the source domain is the same as the set of actions that will be useful in the target domain . Our method does not require these assumptions , allowing it to successfully learn in settings where these prior works would fail . For example , the assumption of Sherstov & Stone ( 2005 ) is violated in our experiments with broken robots : actions which move a joint are useful in the source domain ( where the robot is fully-function ) but not useful in the target domain ( where that joint is disabled ) . Our method will significantly outperform an importance weighting baseline ( Lazaric , 2008 ) . Unlike Vemula et al . ( 2020 ) , our method does not require learning a dynamics model and is applicable to stochastic environments and those with continuous states and actions . Our algorithm bears a resemblance to that in Wulfmeier et al . ( 2017b ) , but a crucial algorithmic difference allows us to prove that our method acquires a near-optimal policy in the target domain , and also leads to improved performance empirically . The theoretical derivation of our method is inspired by prior work which formulates control as a problem of probabilistic inference ( e.g. , ( Toussaint , 2009 ; Rawlik et al. , 2013 ; Levine et al. , 2018 ) ) . Algorithms for model-based RL ( e.g. , ( Deisenroth & Rasmussen , 2011 ; Hafner et al. , 2018 ; Janner et al. , 2019 ) ) and off-policy RL ( e.g. , ( Munos et al. , 2016 ; Fujimoto et al. , 2018 ; Dann et al. , 2014 ; Dudı́k et al. , 2011 ) similarly aim to improve the sample efficiency of RL , but do use the source domain to accelerate learning . Our method is applicable to any maximum entropy RL algorithm , including on-policy ( Song et al. , 2019 ) , off-policy ( Abdolmaleki et al. , 2018 ; Haarnoja et al. , 2018 ) , and model-based ( Janner et al. , 2019 ; Williams et al. , 2015 ) algorithms . We will use the SAC ( Haarnoja et al. , 2018 ) in our experiments and compare against model-based baselines . 3 PRELIMINARIES . In this section , we introduce notation and formally define domain adaptation for RL . Our problem setting will consider two MDPs : Msource represents the source domain ( e.g. , a practice facility , simulator , or learned approximate model of the target domain ) whileMtarget represents a the target domain . We assume that the two domains have the same state space S , action space A , reward function r , and initially state distribution p1 ( s1 ) ; the only difference between the domains is the dynamics , psource ( st+1 | st , at ) and ptarget ( st+1 | st , at ) . We will learn a Markovian policy πθ ( a | s ) , parametrized by θ . Our objective is to learn a policy π that maximizes the expected discounted sum of rewards onMtarget , Eπ , Mtarget [ ∑ t γ tr ( st , at ) ] . We now formally define our problem setting : Definition 1 . Domain Adaptation for RL is the problem of using interactions in the source MDP Msource together with a small number of interactions in the target MDPMtarget to acquire a policy that achieves high reward in the target MDP , Mtarget . We will assume every transition with non-zero probability in the target domain will have non-zero probability in the source domain : ptarget ( st+1 | st , at ) > 0 =⇒ psource ( st+1 | st , at ) > 0 for all st , st+1 ∈ S , at ∈ A . ( 1 ) This assumption is common in work on importance sampling ( Koller & Friedman , 2009 , §12.2.2 ) , and the converse need not hold : transitions that are possible in the source domain need not be possible in the target domain . If this assumption did not hold , then the optimal policy for the target domain might involve behaviors that are not possible in the source domain , so it is unclear how one could learn a near-optimal policy by practicing in the source domain .

Summary: The paper introduces DARC, a domain transfer algorithm motivated by maximum entropy RL. By introducing classifiers for the target and source domain, the reward function in the source domain can be modified such that it restricts the behavior of the optimized policy to transitions that reflect the target domain. In this way, the method achieves good domain transfer without having an explicit model.