Cross Domain Imitation Learning

1 INTRODUCTION . Humans possess an astonishing ability to recognize latent structural similarities between behaviors in related but distinct domains , and learn new skills from cross domain demonstrations alone . Not only are we capable of learning from third person observations that have no obvious correspondence to our internal self representations ( Stadie et al. , 2017 ; Liu et al. , 2018 ; Sermanet et al. , 2018 ) , but we also are capable of imitating agents with different embodiments ( Gupta et al. , 2017 ; Rizzolatti & Craighero , 2004 ) as can be observed in an infant ’ s learning of visuomotor skills from adults with different biomechanics and physical capabilities ( Jones , 2009 ) . Previous work in neuroscience ( Marshall & Meltzoff , 2015 ) and robotics ( Kuniyoshi & Inoue , 1993 ; Kuniyoshi et al. , 1994 ) have recognized the pitfalls of exact behavioral cloning in the presence of domain discrepancies and posited that the effectiveness of the human imitation learning mechanism hinges , crucially , on the capability to learn structure preserving domain correspondences . These correspondences enable the learner to internalize the expert demonstrations and produce a reconstruction of the behavior in the self domain . Consider a young child that has learned to associate his internal body map with the limbs of an adult . When the adult demonstrates running , the child is able to imagine himself running , and reproduce the behavior . Recently , separate solutions have been proposed for imitation learning across two kinds of domain discrepancies : embodiment ( Gupta et al. , 2017 ) and viewpoint ( Liu et al. , 2018 ; Sermanet et al. , 2018 ) mismatch . These works ( Liu et al. , 2018 ; Sermanet et al. , 2018 ; Gupta et al. , 2017 ) require paired , time-aligned demonstrations to obtain state correspondences and an extra RL step with a proxy reward . However , paired , aligned demonstrations are seldom obtainable and RL loops are expensive . In this work we formalize the Cross Domain Imitation Learning ( CDIL ) problem which encompasses prior work in imitation learning across domains with viewpoint and embodiment mismatch . Informally , CDIL is the process of learning how to perform a task optimally in a self domain , given demonstrations of the task in a distinct expert domain . We propose a two-step approach to CDIL : alignment followed by adaptation . In the alignment step we execute a novel unsupervised MDP alignment algorithm , Generative Adversarial MDP Alignment ( GAMA ) , to learn state , action maps from unpaired , unaligned demonstrations . In the adaptation step we leverage the learned state , action maps to zero-shot imitate tasks across domains without an additional RL step . To shed light on when CDIL can be solved by alignment and adaptation , we first introduce a class of structure preserving maps , called MDP reductions , that adapts optimal policies between MDPs ( section 3 ) . We further characterize a family of MDP pairs that share reductions , formally state the MDP alignment problem , and elucidate its connection to CDIL . In section 4 , 5 we derive GAMA , a simple training algorithm to learn MDP reductions . In section 6 , we experimentally evaluate GAMA and find that meaningful state correspondences between various domains are learned from unpaired , unaligned demonstrations . We then compare the CDIL performance of GAMA against several baselines in both embodiment and viewpoint mismatch scenarios and show the effectiveness of our approach . 2 CROSS DOMAIN IMITATION LEARNING PROBLEM STATEMENT . An infinite horizon Markov Decision Process ( MDP ) M 2 ⌦ with deterministic dynamics is a tuple ( S , A , P , ⌘ , R ) where ⌦ is the set of all MDPs , S is the state space , A is the action space , P : S⇥A ! S is a ( deterministic ) transition function , R : S ⇥ A ! R is the reward function , and ⌘ is the initial state distribution . A domain is an MDP without the reward , i.e ( S , A , P , ⌘ ) . Intuitively , a domain fully characterizes the embodied agent and the environment dynamics , but not the desired behavior . A task T is a label for an MDP corresponding to the high level description of optimal behavior , such as `` walking '' . T is analogous to category labels for images . An MDP with domain x for task T is denoted by Mx , T = ( Sx , Ax , Px , ⌘x , Rx , T ) , where Rx , T is a reward function encapsulating the behavior labeled by T . For example , different reward functions are needed to realize the `` walking '' behavior in two morphologically different humanoids . A ( stationary ) policy for Mx , T is a map ⇡x , T : Sx ! B ( Ax ) where B is the set of probability measures on Ax and an optimal policy ⇡⇤x , T = argmax⇡x J ( ⇡x ) achieves the highest policy performance J ( ⇡x ) = E⇡x [ P1 t=0 tRx , T ( s ( t ) x , a ( t ) x ) ] where 0 < < 1 is a discount factor . A demonstration of length H is a sequence of state , action tuples ⌧Mx , T = { ( s ( t ) x , a ( t ) x ) } Ht=1 sampled from an optimal policy and DMx , T = { ⌧ ( k ) Mx , T } K k=1 is a set of demonstrations for Mx , T Let Mx , T , My , T be self and expert MDPs for a target task T . Given expert domain demonstrations DMy , T , Cross Domain Imitation Learning ( CDIL ) aims to determine an optimal self domain policy ⇡ ⇤ x , T without access to the reward function Rx , T . In this work we propose to first solve an MDP alignment problem and then leverage the alignments to zero-shot imitate expert domain demonstrations . Like prior work ( Gupta et al. , 2017 ; Liu et al. , 2018 ; Sermanet et al. , 2018 ) , we assume the availability of an alignment task set Dx , y = { ( DMx , Ti , DMy , Ti ) } N i=1 containing demonstrations for N tasks { Ti } Ni=1 from both the self and expert domain . Dx , y could , for example , contain both robot ( x ) and human ( y ) demonstrations for a set primitive tasks such as walking , running , and jumping . Unlike prior work , demonstrations are unpaired and unaligned , i.e ( s ( t ) x , s ( t ) y ) may not be a valid state correspondence . ( see Figure 1 ( a ) ) Paired , time-aligned cross domain data is expensive and may not even exist when task execution rates differ or there exists systematic embodiment mismatch between the domains . For example , a child can imitate an adult running , but not achieve the same speed . Our set up emulates a natural setting in which humans compare how they perform tasks to how other agents perform the same tasks in order to find structural similarities and identify domain correspondences . We now proceed to introduce a theoretical framework that explains how and when the CDIL problem can be solved by MDP alignment followed by adaptation . 3 ALIGNABLE MDPS . Let ⇧⇤M be the set of all optimal policies for MDP M. We define an occupancy measure ( Syed et al. , 2008 ) q⇡ : S ⇥A ! R for policy ⇡ as q⇡ ( s , a ) = ⇡ ( a|s ) P1 t=0 t Pr ( s ( t ) = s ; ⇡ , P , ⌘ ) . Definition 1 . An optimality function OMx : Sx ⇥ Ax ! { 0 , 1 } for an MDP Mx satisfies : OMx ( sx , ax ) = 1 if 9⇡⇤x 2 ⇧⇤Mx such that ( sx , ax ) 2 supp ( q⇡⇤x ) and OMx ( sx , ax ) = 0 otherwise . Definition 2 . An MDP reduction from Mx = ( Sx , Ax , Px , ⌘x , Rx ) to My = ( Sy , Ay , Py , ⌘y , Ry ) is a tuple r = ( , ) where : Sx ! Sy , : Ax ! Ay are maps that preserve : 1 . ( optimal policy ) 8sx 2 Sx , ax 2 Ax , sy 2 Sy , ay 2 Ay , OMy ( ( sx ) , ( ax ) ) = 1 ) OMx ( sx , ax ) = 1 ( 1 ) OMy ( sy , ay ) = 1 ) 1 ( sy ) 6= ; , 1 ( ay ) 6= ; ( 2 ) 2 . ( dynamics ) 8sy , s0y 2 Sy , ay 2 Ay where OMy ( sy , ay ) = 1 , Py ( sy , ay ) = ( Px ( sx , ax ) ) 8sx 2 1 ( sy ) , ax 2 1 ( ay ) ( 3 ) where we define 1 ( sy ) = { sx| ( sx ) = sy } , 1 ( ay ) = { ax| ( ax ) = ay } . Furthermore , r is an MDP permutation if and only if , are bijective maps . In words , Eq . 1 states that only optimal state , action pairs in x can be mapped to optimal state , action pairs in y and Eq . 2 states that r must be surjective on the set of optimal state , action pairs in y. Eq . 3 states that a reduction must preserve ( deterministic ) dynamics . We use the notation Mx , My to denote that ( , ) is a reduction from Mx to My , and the shorthand Mx My to denote that Mx reduces to My . To gain an intuitive understanding of MDP reductions , picture the execution trace of an optimal policy as a directed graph with colored edges in which the nodes correspond to states visited by an optimal policy , and the colored edges correspond to actions taken . An MDP reduction from Mx to My homomorphs the execution graph of an optimal policy in Mx to a execution graph of an optimal policy in My . Figure 2 shows an example of a valid reduction from Mx to My : states 1 , 2 in Sx are mapped ( merged ) to state a in Sy and the blue , green actions in Ax are mapped to the brown action in Ay . Intuitively , if Mx , My , then ( , ) compresses Mx by merging all optimal state , action pairs that have identical dynamics properties . Definition 3 . Two MDPs Mx , My are alignable if and only if Mx My or My Mx . Definition 3 states that MDPs are alignable if reductions exists between them , meaning that they share structure . We use ( Mx , My ) = { ( , ) |Mx , My } to denote the set of all valid reductions from Mx to My . Reductions have a particularly useful property which is that they adapt policies across alignable MDPs . Consider a state map f : Sx ! Sy , an inverse action map g : Ay ! Ax , and a composite policy ⇡̂x = g ⇡y f ( see Figure 1 ( b ) ) . In words , ⇡̂x maps a self state to an expert state via f , simulates the expert ’ s action choice for the mapped state via ⇡y , then chooses a self action that corresponds to the simulated expert action with g. The following lemma holds for ⇡̂x . Lemma 1 . Let Mx , My be MDPs satisfying Assumption 1 ( see Supp . Materials ) , Mx , My , and ⇡y be optimal in My . 8g : Ay ! Ax s.t g ( ay ) = ay 8ay 2 { ay|9sy 2 Sy s.t OMy ( sy , ay ) = 1 } , it holds that ⇡̂x = g ⇡y is optimal in Mx . Lemma 1 states that the state , action maps ( f , g 1 ) chosen to be a reduction can adapt optimal policies between alignable MDPs . Here onwards we interchangeably refer to ( f , g ) as `` alignments '' . We now show how the CDIL problem can be solved by first solving an MDP alignment problem followed by an adaptation step . Definition 4 . Let ( Mx , My ) , ( Mx0 , My 0 ) 2 ⌦2 be two MDP pairs . Then , ( Mx , My ) ⇠ ( Mx0 , My 0 ) , i.e they are joint alignable , if and only if ( Mx , My ) \ ( Mx0 , My 0 ) 6= ; . In words , two MDP pairs are joint alignable if there exists a shared reduction . We define an equivalence class [ ( Mx , My ) ] ⇠ = { ( Mx0 , My 0 ) | ( Mx0 , My 0 ) ⇠ ( Mx , My ) } of MDP pairs that share reductions . Overloading notation , ( { ( Mxi , Myi ) } Ni=1 ) = { ( , ) | ( , ) 2 ( Mx 1 , Mx1 ) \ · · · \ ( MxN , MxN ) } . We now formally state the MDP alignment problem : Let ( Mx , T , My , T ) be an MDP pair for a target task T . Given an alignment task set Dx , y = { ( DMx , Ti , DMy , Ti ) } N i=1 comprising unpaired , unaligned demonstrations for MDP pairs { ( Mx , Ti , My , Ti ) } N i=1 ✓ [ ( Mx , T , My , T ) ] ⇠ , determine ( , ) 2 ( { ( Mx , Ti , My , Ti ) } N i=1 ) such that ( , ) 2 ( Mx , T , My , T ) . As shown in Figure 3 , with more MDP pairs , there are likely a smaller the number of joint alignments | ( { ( Mx , Ti , My , Ti ) } N i=1 ) | and , as a result , ( , ) 2 ( { ( Mx , Ti , My , Ti ) } N i=1 ) is more likely to `` generalize '' to an MDP pair for a new target task ( Mx , T , My , T ) in the equivalence class . Analogously , in a standard supervised learning problem , more training data is likely to shrink the set of models performing optimally on the training set but poorly on the test set . We can then use ( , ) for CDIL : given cross domain demonstrations DMy , T for the target task T , learn an expert domain policy ⇡y , T , and adapt it into the self domain using ( , ) according to Lemma 1 . We can now assess when domains with embodiment and viewpoint mismatch have meaningful state correspondences , i.e MDP reductions , thus allowing for cross domain imitation . The states of a human expert with more degrees of freedom than a robot imitator can be merged into the robot states if the task only requires the robot ’ s degrees of freedom and the execution traces share structure , e.g traces are both cycles . However , if the task requires all degrees of freedom possessed only by the human , the robot can not find meaningful correspondences , and also can not imitate the task . Two MDPs for different viewpoints of an agent performing a task are MDP permutations since there is a one-to-one correspondence between state , actions at same timestep in the execution trace of an optimal policy .

Summary: This paper proposes Generative Adversarial MDP Alignment (GAMA) for imitation learning. Given a set of paired MDPs, GAMA learns a state mapping f and an action mapping g such that one MDP can be reduced to another. For a new test MDP pair (x,y) where expert demonstrations are available for y, GAMA can use f to map a state of x to a state of y, mimic the expert behavior, then use g to map the expert action back to an action in x. The reduction is theoretically motivated, the optimization is based on adversarial learning and finally, experiments on common Gym environments show that GAMA can achieve effective transfer.