Cross Domain Imitation Learning
Authors: Kun Ho Kim
Yihong Gu
Jiaming Song
Shengjia Zhao
Stefano Ermon
Venue:  (2020)
Keywords: Imitation Learning
Domain Adaptation
Reinforcement Learning
Zeroshot Learning
Machine Learning
Artificial Intelligence

Abstract: We study the question of how to imitate tasks across domains with discrepancies such as embodiment and viewpoint mismatch. Many prior works require paired, aligned demonstrations and an additional RL procedure for the task. However, paired, aligned demonstrations are seldom obtainable and RL procedures are expensive. In this work, we formalize the Cross Domain Imitation Learning (CDIL) problem, which encompasses imitation learning in the presence of viewpoint and embodiment mismatch. Informally, CDIL is the process of learning how to perform a task optimally, given demonstrations of the task in a distinct domain. We propose a two step approach to CDIL: alignment followed by adaptation. In the alignment step we execute a novel unsupervised MDP alignment algorithm, Generative Adversarial MDP Alignment (GAMA), to learn state and action correspondences from unpaired, unaligned demonstrations. In the adaptation step we leverage the correspondences to zero-shot imitate tasks across domains. To describe when CDIL is feasible via alignment and adaptation, we introduce a theory of MDP alignability. We experimentally evaluate GAMA against baselines in both embodiment and viewpoint mismatch scenarios where aligned demonstrations don’t exist and show the effectiveness of our approach.

Under review as a conference paper at ICLR 2020
CROSS DOMAIN IMITATION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
We study the question of how to imitate tasks across domains with discrepancies such as embodiment and viewpoint mismatch. Many prior works require paired, aligned demonstrations and an additional RI. procedure for the task. However, paired, aligned demonstrations are seldom obtainable and RL procedures are expensive. In this work, we formalize the Cross Domain Imitation Learning (CDIL) problem, which encom- passes imitation learning in the presence of viewpoint and embodiment mismatch. Informally, CDIL is the process of learning how to perform a task optimally, given demonstrations of the task in a distinct domain. We propose two step approach to CDIL: alignment followed by adaptation. In the alignment step we execute a novel unsu- pervised MDP alignment algorithm, Generative Adversarial MDP Alignment (GAMA), to learn state and action correspondences from unpaired, unaligned demonstrations. In the adaptation step we leverage the correspondences to zero-shot imitate tasks across domains. To describe when CDIL is feasible via alignment and adaptation, we introduce a theory of MDP alignability. We experimentally evaluate GAMA against baselines in both embodiment and viewpoint mismatch scenarios where aligned demonstrations don't 't exist and show the effectiveness of our approach.
INTRODUCTION
Humans possess an astonishing ability recognize latent structural similarities between behaviors in related but distinct domains, and learn new skills from cross domain demonstrations alone. Not only are we capable of learning from third person observations that have no obvious correspondence to our internal self representations (Stadie et al., 2017: Liu et al., 2018; Sermanet et al., 2018), but we also are capable of imitating agents with different embodiments (Gupta et al., 2017: Rizzolatti & Craighero, 2004) as can be observed in infant's 's learning of visuomotor skills from adults with different biomechanics and physical capabilities (Jones, 2009). Previous work in neuroscience (Marshall & Meltzoff, 2015) and roboties (Kuniyoshi & Inoue, 1993; Kuniyoshi al., 1994) have recognized the pitfalls of exact behavioral cloning in the presence of domain discrepancies and posited that the effectiveness of the human imitation learning mechanism hinges, crucially, on the capability to learn structure preserving domain correspondences. These correspondences enable the learner to internalize the expert demonstrations and produce reconstruction of the behavior in the self domain. Consider young child that has learned to associate his internal body map with the limbs of an adult. When the adult demonstrates running, the child is able to imagine himself running, and reproduce the behavior.
Recently, separate solutions have been proposed for imitation learning across two kinds of domain discrepancies: embodiment (Gupta et al., 2017) and viewpoint (Liu et al., 2018; Sermanet et al., 2018) mismatch. These works (Liu et al., 2018; Sermanet et al., 2018; Gupta et al., 2017) require paired, time-aligned demonstrations to obtain state correspondences and an extra RL step with a proxy reward. However, paired, aligned demonstrations are seldom obtainable and RL loops are expensive. In this work we formalize the Cross Domain Imitation Learning (CDIL) problem which encompasses prior work in imitation learning across domains with viewpoint and embodiment mismatch. Informally, CDIL is the process of learning how to perform a task optimally in a self domain, given demonstrations of the task in a distinct experi domain. We propose two-step approach to CDIL: alignment followed by adaptation. In the alignment step we execute a novel unsupervised MDP alignment algorithm. Generative Adversarial MDP Alignment (GAMA), to learn state, action maps from unpaired, unaligned demonstrations. In the adaptation step we leverage the learned state, action maps to zero-shot imitate tasks across domains without an additional RL step. To shed light on when CDIL can be solved by alignment and adaptation, we first introduce a class of structure preserving maps, called MDP reductions, that adapts optimal policies between MDPs (section 3). We further characterize a family of MDP pairs that share reductions, formally state the MDP alignment problem, and elucidate its connection to CDIL. In section 4, 5 we derive GAMA,

---

Under review as a conference paper at ICLR 2020
a
Alignment
Unpaired Unaiigned Day
C.
Adaptation
Paired Aligned Dxy
(Sys By Sy), (5pz Ggy $/)
11121
T
Da. Dist. Match min d(e""" ,et)
y
3/2
F-88
fa
Unaligned
Umpaired
J
Dema Aclast
Dzy
B.C Imitation min (+G181)))
B.G
F2
Folicy
R1.7 ROT
Adopl
Figure 1: (a). Illustration of paired, aligned unpaired, unaligned demonstrations in the alignment task set Da.yy (b). Alignment: we learn state, action maps f. between the self (z) and expert (y) domain from unpaired, unaligned demonstrations by minimizing distribution matching loss and imitation loss. (c) Adaptation: adapt the expert domain policy Ty. or demonstrations to obtain a self domain policy я,,т
simple training algorithm learn MDP reductions. In section 6, we experimentally evaluate GAMA and find that meaningful state correspondences between various domains are learned from unpaired, unaligned demonstrations. We then compare the CDIL performance of GAMA against several baselines in both embodiment and viewpoint mismatch scenarios and show the effectiveness of our approach.
2 CROSS DOMAIN IMITATION LEARNING PROBLEM STATEMENT
An infinite horizon Markov Decision Process (MDP) M E Q with deterministic dynamics is a tuple (S,A,
P, A,,AA,P, is (deterministic) transition function, R S x A + R is the reward function, and 17 is the initial state distribution. A domain is an MDP without the reward, i.e (S, P.n). Intuitively, domain fully characterizes the embodied agent and the environment dynamics, but not the desired behavior. A task T is a label for an MDP corresponding the high level description of optimal behavior, such as "walking". " T is analogous to category labels for images. An MDP with domain x for task T is denoted by M2. T = Age Pz. nu. RR.T). where R2,T is reward function encapsulating the behavior labeled by T. For example, different reward functions are needed to realize the "walking" behavior in two morphologically different humanoids. A (stationary) policy for MM.T is a map Nz,T : S, + ((A..) where B is the set of probability measures on Ar and an optimal policy Z²,T = arg max [(T.)) achieves
R) where 5 is the set of all MDPs, S is the state space, A the action space, P 8 x A + S
the highest policy performance J ((TT) = ET, IEi RR (s.") aa²)) where 0 < 7 < 1 is discount factor. A demonstration of length H is a sequence of state, action tuples TM..T = {(5 az )E
(8) sampled from an optimal policy and DM... = { 1 is a set of demonstrations for M2,T
Let Me,T, My,T be self and expert MDPs for target task T. Given expert domain demonstrations D DM,, Cross Domain Imitation Learning (CDIL) aims to determine an optimal self domain policy T., without access to the reward function Rr, In this work we propose to first solve an MDP alignment problem and then leverage the alignments to zero-shot imitate expert domain demonstrations. Like prior work (Gupta et al., 2017: Liu et al., 2018; Sermanet et al., 2018), we assume the availability of an alignment task set Dr.y = ((DM,, T. DM,,, T, ) containing demonstrations for N tasks (Tr) from both the self and expert domain. Da.y, could, for example, contain both robot (r) and human (v) demonstrations for a set primitive tasks such as walking, running, and jumping. Unlike prior work, demonstrations are
(t) may not be valid state correspondence. (see Figure 1(a)) Paired, time-aligned cross domain data is expensive and may not even exist when task execution rates differ or there exists systematic embodiment mismatch between the domains. For example, child can imitate adult running, but not achieve the same speed. Our set up emulates natural setting in which humans compare how they perform tasks to how other agents perform the same tasks in order to find structural similarities and identify domain correspondences. We now proceed to introduce a thcoretical framework that explains how and when the CDIL problem can be solved by MDP alignment followed by adaptation.
umpaired and unaligned, i.e (s% Sy
3 ALIGNABLE MDPs
Let 11,, be the set of all optimal policies for MDP M. We define an occupancy measure (Syed et al., 2008) 4x S x A + RR for policy x as 9x (s, a) = T(ass) Ei 7a Pr(((t) = т. P.n). Definition 1. An optimality function OM, : S2 x A + {0.1] for an MDP M₂ satisfies:
OM, (8z, az)) = 1 if 3m E 11' such that (Sa, ax) E supp(gn;) and OM (sn,ax) = 0 otherwise.

---

Under review as a conference paper at ICLR 2020
Definition 2. An MDP reduction from M₁ AS, P2: nu. Ru) My = (Sy. Ay, Py. Fy Ruy) is
tuple r = (ó. v) where @: p Sz + Sy. v. Az + Ay are maps that preserve:
1. (optimal policy) Vsz E ES2,0z E Ay, E Sy, ay E Ay
OM, (0(52)
(xx))) (((())))((a)))))
=1 7
(8z, Az) =1
(1)
OM, OM, (sy, ay) =1
d-y # -,,, -1010(((a)) # 0
(2)
2. (dynamics) Vsy, 8y E Sy ay E Ag where OM, (8g, (8y: =1,
(8y); O2 E E (ag) where we define &- (s,) = {82 (5x])(5s)) = =yy & 1 (ay) = {ax = ay}. Furthermore, r is an MDP permutation and only if o, to are bijective maps.
= 2 (P((0n) (z)) V8, E Ex = ((8y),
In words, Eq. states that only optimal state, action pairs in x can be mapped to optimal state, action pairs in y and Eq. 2 states that must be surjective on the set of optimal state, action pairs in y. Eq. 3 states that a reduction must preserve (deterministic) dynamics. We use the notation Ma Zo. My to denote that (ø, vb) is a reduction from MM, to My. and the shorthand M2 2 My to denote that M2 reduces to My. To gain an intuitive understanding of MDP reductions, picture the execution trace of an optimal policy as a directed graph with colored edges in which the nodes correspond to states visited by an optimal policy, and the colored edges correspond to actions taken. An MDP reduction from M2 to My homomorphs the execution graph of an optimal policy in M, to execution graph of an optimal policy in My. Figure shows an example of a valid reduction from M2 to My: states 1.2 in S2 are mapped (merged) to state in Sy and the blue, green actions in Ag are mapped to the brown action in Ay. Intuitively, if M₂ Z My. then (ø, e) compresses Mg by merging all optimal state, action pairs that have identical dynamics properties.
(11 = (2) a,
M2
2
3
(33 =
(4) = C
(55) = d
Mr
3 5
(6)
V = 4
Figure 2: Example MDP reduction from M₂ to My. o, d are state and action maps
Definition 3. Two MDPs Ma, My are alignable if and only if M2 2 My or My 2 Ma.
Definition 3 states that MDPs are alignable if reductions exists between them, meaning that they share structure. We use r'M,, My)) = {(o, M) ZΦ.0 My to denote the set of all valid reductions from M2 to My. Reductions have a particularly useful property which is that they adapt policies across alignable MDPs. Consider state map f Sy + Sy. an inverse action map g Ay A.. and composite policy Ω = g x o f (see Figure ((b).. In words, #₂ maps self state to an expert state via f. simulates the expert's action choice for the mapped state via Ty. then chooses a self action that corresponds to the simulated expert action with g. The following lemma holds for 7..
Lemma 1. Let Ma, Ms,My be MDPs satisfying Assumption I (see Supp. Materials), Mr Zo.0 My and Ty be optimal in My- Vg : Ay + Ar s.t Ao g(ay) By Vay E {ayl3sy E Sy s.t Om, ag) 1}. it holds that म.a g o Ty o @ is optimal in Mz.
Lemma 1 states that the state, action maps (J,g=1) chosen to be reduction can adapt optimal policies between alignable MDPs. Here onwards we interchangeably refer to (S.g) as "alignments". We now show how the CDIL problem can be solved by first solving an MDP alignment problem followed by adaptation step.
Definition 4. Let (Mz, (My', M,,' E Ω be two MDP pairs. Then, (Ms, ~ (Ma's MMy',My), ie they are joint alignable, if and only ifT(M,, N(M,'' P(MM',,,, ,My') + 0.
In words, two MDP pairs are joint alignable if there exists shared reduction. We define equiva- (Mz',,y') ~ (M)) of MDP pairs that share
Ience class [(MM, = ((ML', Myy), reductions. Overloading notation, T([(M2 & MM () = ((0, v)) (Ø) v) E T(M,* 2 n... T(M2 N (M, We now formally state the MDP alignment problem: Let Nee,T,,,,r be an MDP pair for a target task T. Given an alignment task set Da..y = ((DMo.T) DM, yN compris- ing unpaired, unaligned demonstrations for MDP pairs {(Me.T My.T.. S (M,M,TTM,TTTi. determine (ø, v) E I(f(M2,T, TT, My.T, MM, such that (ò, ()) E (MM.TT.M.T.. As shown in Figure 3, with more MDP pairs, there are likely a smaller the number of joint align- is more likely to "generalize" to an MDP pair for a new target task (Ma,T, in the equivalence class. Analogously, in a standard supervised learning problem, more training data is likely to shrink the set of models performing optimally on the training set but poorly on the test set.
ments (T([MM,,T; M,,T, )), and, as a result, (0. E MH
3

---

Under review as a conference paper at ICLR 2020
We can then use (Ø, 1b) for CDIL: given cross domain demonstrations D.M..r for the target task T, learn expert domain policy Ty.T, and adapt it into the self domain using (Ø, x according to Lemma
Alggmmet
Tails
We can now assess when domains with embodiment and viewpoint mismatch have meaningful state correspondences, i.e MDP reductions, thus allowing for cross domain imitation. The states of a human expert with more degrees of freedom than a robot imitator can be merged into the robot states if the task only requires the robot's 's degrees of freedom and the execution traces share structure, e-g traces are both cycles. However, if the task requires all degrees of freedom possessed only by the human, the robot cannot find meaningful correspondences, and also cannot imitate the task. Two MDPs for different viewpoints of an agent performing a task are MDP permutations since there is a one-to-one correspondence between state, actions at same timestep in the execution trace of an optimal policy.
Wil MaghlMaa Maaill
Target Tank
Illustration of MDP alignment problem
Figure 3:
4 LEARNING MDP REDUCTIONS
We now derive objectives that can be optimized to learn MDP reductions. We propose distribution matching and policy performance maximization. We first define the distributions to be matched.
Definition 5. Let Mg. My be two MDPs and it = g a Ty o f for f S2 + Sy : Ay + Ag and
(() policy T.. P = (1) By 1r20 is the co-domain policy execution process realized by rumning at.. i.e:
12 a(t) y f(s ) a Ty (-18 ) = g(aaf)) s s((+1) Ps(
V2 0 (4)
The target distribution oW, over transitions uniformly sampled from execution traces of Ty and the proxy distribution of オーサ is over cross domain transitions uniformly sampled from realizations P.
". (By. Gy. (y)) = lim 1
T-
Pr(s
(t) Syy ay &yy (t-1) 8/y Ty. Pa. 17)
(5)
T
T
z (sy, x))) = lim T- T
1
(t) Eis Pr(s (t) = Sy, dg By By
(t=1) P)
(6)
We now propose three concrete objectives: 1. मे is optimal, 2. 2ール 0%, 3. g is injective. In other words, seek learn f,g that matches distributions over transition tuples in domain y while maximizing policy performance in domain x. The former captures the dynamics preservation property from Eq. 3 and the latter captures the optimal policy preservation property from Eq. 1, 2. The following theorem uncovers the connection between our objectives and MDP reductions.
Theorem 1. Let My be MDPs satisfying Assumption (see Supp Materials). Mf 2 My. then S Sy. Ay Aa, and optimal covering policy Ny (Supp Materials, Def 6) that satisfy objectives 1, 2. Conversely, ff :S - Syy :A, + Ag and an optimal covering policy Ty satisfying objectives 1, 2, 3, then M Z My and 3(0, 1) 11(M,, s.1 à and oo vog(a,) =ay Vay E Ay-
Theorem 1 states that if two MDP are alignable, then objectives 1, 2 can be satisfied. Conversely, if objectives 1, 2, 3 can be satisfied for two MDPs, then they must be alignable and all solutions (f,g) are MDP reductions. While Theorem requires alignable MDPs to guarantee identifiability, our experiments will also run on MDPs that are not perfectly alignable, i.e. Eq. 1, 2, 3 do not hold exactly, but intuitively share structure. In the next section, we propose a simple algorithm to learn MDP reductions.
5 GENERATIVE ADVERSARIAL MDP ALIGNMENT
Building on Theorem 1, we propose the following general form training objective for aligning MDPs:
min (((tt) Ad(o 0 f.1
where J(Hz) is the performance of S.. d is distance metric between distributions, and À > 0 is Lagrange multiplier. practice, we found that injectivity of g is unnecessary to enforce in continuous domains. We now present an instantiation of this framework: Generative Adversarial MDP Alignment (GAMA). Recall that we are given an alignment task set Da.y = {(DMa n. DM, ) the alignment step, we learn "g T. VT, and parameterized state, action maps fe, S2 + Sy. ge, A. + Ag that compose it. 90, T.T. T. foy- To match a -- o employ adversarial training (Goodfellow

---

Under review as a conference paper at ICLR 2020
et al., 2014) in which separate discriminators Døj per task are trained to distinguish between "real" transitions (5y, ays syy) ~ T.. and "fake" transitions (sy: 8y) R.. where 8, fo, (52), âg = fitted model of the x domain dynamics. (see Figure
Ty (y = fo, (Poj ( g(g)))) , and Pop 1(b)) The generator, consisting of fo,: -ng is trained to fool the discriminator while maximizing policy performance. The distribution matching gradients are back propagated through the learned dynamics, ".. is learned by Imitation Learing (II.) on DM,,7," and the policy performance objective on Rx,7, is achieved by IL on PM..T." In this work we use behavioral cloning (Pomerleau, 1991) for IL. We thus seek to find a saddle point {f.g} U (Do Jimz of the following objective:
min max Ei= (Ba,nai.7, ( (-18x)) ( (-18=))] fig (Dog 12
(8)
+ *BBrg, [log Døg (sy, Gzy s'y) Bx [log(1 Døg (3y, Oy ()))
where DEL is the KL--ivergence. We provide the full execution flow of GAMA in Algorithm 1 In the adaptation step, we are given expert demonstrations DM,,, of a new target task T, from which we fit an expert domain policy T.,T which are composed with the learned alignments to construct an adapted self policy $2,7 90, Ty.7 foy. We also experiment with demonstration adaptation method which additionally trains an inverse state map s-1 : S, Sg,, adapts demonstrations D DM,,7 into the self domain via f 9.. and applies behavioral cloning on the adapted demonstrations. (see Figure I(c) Notably, our entire procedure does not require paired, aligned demonstrations nor an RL step.
Algorithm 1: Generative Adversarial MDP Alignment (GAMA)
input: Alignment task set Dz,y= = (DDMe,T, DMg, m. M of unpaired trajectories, fitted T
while not done do:
for i = I.., N do:
Sample (8z, 0z2 5/2) ~ D DM v (8y: ayy 8'y) D DM. and store in buffer By B
for ju 1, ..., M do:
Sample mini-batch j from Bi Bi
Update dynamics model with: - -B Ver (Pop (80, (x) - -3
Update discriminator: B (Vo log Doj, (sy: By s/y)] ) B Voo log (1 Dos, (8y, au 81))
Update alignments (Say,g0,) with gradients:
E E .T. Vo, log Do (8y, ay 8y)1 Er Vo, (fn,T, (82) - -az)21
- -Em; T. Vo, log DoD (By, ay 8,)] E (Ta,T - -az) )x)
Closely related to CDIL, the field of cross domain transfer learning in the context of RL has explored approaches to use state maps to exploit cross domain demonstrations in pretraining procedure for a new target task for which self domain reward function is available. Canonical Corre- lation Analysis (CCA) (Hotelling, 1936) finds invertible projections into a basis in which data from different domains are maximally correlated. These projections can then be composed to obtain a direct correspondence map between states. Ammar et al. (2015): Joshi & Chowdhary (2018) have utilized an unsupervised manifold alignment (UMA) algorithm which finds linear map between states with similar local geometric properties. UMA assumes the existence of hand crafted features along with distance metric between them. This family of work commonly uses a linear statemap to define a time-step wise transfer reward and executes an RL step on the new task. Similar to our work, these works use an alignment task set of unpaired, unaligned trajectories to compute the state map. Unlike these works, we learn maps that preserve MDP structure, use deep neural network state, action maps, and achieve zero-shot transfer to the new task without an RL step. More recent work in transfer learning across embodiment (Gupta et al., 2017) and viewpoint (Liu et al., 2018; Sermanet et al., 2018) mismatch obtain state correspondences from an alignment task set comprising paired, time-aligned demonstrations and use them to learn a state map or state encoder to a domain invariant feature space. In contrast to this family of prior work, our approach learns both state, action maps from umpaired, unaligned demonstrations. Also, we remove the need for additional environment interactions and an expensive RL procedure on the target task by leveraging the action map for zero-shot imitation. Stadie et al. (2017) have shown promise in using domain confusion loss and generative adversarial imitation learning (Ho & Ermon, 2016) for learning across small viewpoint mismatch without an alignment task set, but fails dealing with large viewpoint differences. Unlike Stadie et al. (2017), we leverage the alignment task set to succeed in imitating across
Related Works:
5

---

Under review as a conference paper at ICLR 2020
larger viewpoint mismatch and do not require an RL procedure. MDP homomorphisms (Ravindran & Barto, 2002) have been explored with the aim of compressing state, action spaces to facilitate planning. In similar vein, related works have proposed MDP similarity metrics based on bisimulation methods (Ferns et al., 2004) and Boltzmann machine reconstruction error (Ammar et al., 2014). While conceptually related to our MDP alignability theory, these works have not proposed scalable procedures to discover the homomorphisms and have not drawn connections to cross domain learning.
EXPERIMENTS
Ours experiments were designed to answer the following questions: (1). Can GAMA uncover MDP reductions? (2). Can the learned aligmments (fo,, 90,) be leveraged to succeed CDIL? ? Note that we include experiments with MDP pairs that are not perfectly alignable, yet intuitively share structure, to show general applicabilitty of GAMA for CDIL. We propose three metrics to evaluate the effectiveness of GAMA. First, alignment complexity which is the number of MDP pairs, i.e number of tasks, in the alignment task set needed learn alignments that enable zero-shot imitation, given ample cross domain demonstrations for the target tasks. Second, adaptation complexity which is the amount of cross domain demonstrations for the target tasks needed to successfully imitate tasks in the self domain without querying the target task reward function, given a sufficiently large alignment task set. Finally, transferabiliry, which the environment sample complexity on the target task when using the alignment procedure as weight initialization then running RL with the target task reward function. While we aim to learn optimal self policies without querying the self domain reward function, this metric measures the usefulness of the alignment step even when MDP pairs in the alignment task set are not in the equivalence class of the target MDP pair. We study two ablations of GAMA and compare against the following baselines:
GAMA Policy Adapt (GAMA-PA): learns alignments by Algorithm fits an expert policy Ty.T to DMy,r for a new target task T and zero-shot adapts Ty. to the self domain via मश. = ge, o Ty.T o foz.
GAMA Demonstration Adapt (GAMA-DA): trains f in addition to Algorithm 1, adapts D.M
into the self domain via (f-1, .)). and fits a self domain policy on the adapted demonstrations.
Self Demonstrations (Self-Demo): We behavioral clone self domain demonstrations for the target task. This baseline provides an "upper bound" on the adapation complexity of CDIL.
Canonical Correlation Analysis (Hotelling, 1936) (CCA): finds invertible matrices Cz. Cy to a basis where domain data are maximally correlated from unpaired, unaligned demonstrations.
Unsupervised Manifold Alignment (Ammar et al., 2015) (UMA): finds map between states that have similar local geometries from unpaired, unaligned demonstrations.
Invariant Features (Gupta et al., 2017) (IF): finds invertible projections onto a feature space given state pairings. Dynamic Time Warping (Muller, 2007) is used to obtain the pairings.
Imitation from Observation (Liu et al., 2018) (IfO): learns statemap conditioned
cross domain
observation given state pairings. Dynamic Time Warping (Muller, 2007) is used to obtain the pairings.
Third Person Imitation Learning (Stadie et al., 2017) (TPIL): simultaneously learns domain agnostic feature space and matches distributions in the feature space via GAIL (Ho & Ermon, 2016).
We experiment with environments which are extensions of OpenAl Gym (Brockman et al., 2016). pen, cart, reacher2, reacher3, reach2-tp, snake3, and snake4 denotes the pendulum, cartpole, 2-link reacher, 3-link reacher, third person 2-link reacher, 3-link snake, and 4-link snake environments, respectively. (self domain) + (expert domain) specify an MDP pair in the alignment task set. Model architectures and environment details are further described in the Supp. Materials, section B, C. D.
6.1 MDP ALIGNMENT EVALUATION
Figure 4 visualizes the learned state map fa, for several MDP pairs. The pen <> pen alignment task (Figure Top Left) and reacht-reach-Ip task task exemplify scenarios where two MDPs are permutations of each other. Similarly, the pen < cart aligmment task (Figure 4, Top Right) has reduction that maps the pendulum's 's angle and angular velocity to those of the pole, as the cart's position and velocity are redundant state dimensions once an optimal policy has been learned. Table presents quantitative evaluations of these simple alignment maps. For pen<>pen and reach2+-rreach-1pp we record the average L.2 loss between the learned statemap's 's outputs and the ground truth permutation map's 's outputs. As for pent+cart, we do

---

Under review as a conference paper at ICLR 2020
Self State
1
p
4
GAMA (Ours)
1
CCA
x
2
p
b
b
b
b
UMA
b
IF
p
State
GAMA (Ours)
ZIDDED CORDD
U
IF
Figure Visualization of the learned state maps for pen+>pen (Top Left), pent>cart (Top Right), snake4<>snake3 (Bottom Left), reach2+reech3 (Bottom Right). GAMA is able to recover MDP reduc- tions (Top Left/Righ) and finds interpretable correspondences between domains that are not perfectly alignable, yet intuitively share structure (Bottom Left/Right). Baselines fail in most cases
Table 1: Quantitative evaluation of learned state maps. GAMA reliably finds MDP permutations while baselines incur 10x larger deviation loss from the ground truth permutation map. Error bars/regions show the standard deviation over 5 runs.
GAMA (ours)
CCA
UMA
IrO
Random
pen +> pen
0.057 + 0.017
0.72 = 0.25
>100
2.50
1.08
2.24
0.82
>100
pen cart
0.178 1 0.051
3.92 = 3.77
>100
1.62 + 0.52
3.31 = 1.2
>10
reach2
0.092 1 0.043
10.14 1 5.31
>100
12.41
3.12
5.14
>100
the same on the dimensions that correspond to the angle and angular velocity of the pole. We see from both Figure and Table that GAMA is able to learn simple reductions while baselines fail to do so. The key reason behind this performance gap is that most baselines (Gupta et al., 2017; Liu et al., 2018) obtain state maps from time-aligned demonstration data. However, the considered aligmment task set contains unaligned demonstrations with diverse starting states, up to 2x differences in demonstration lengths, and varying task execution rates. We see that GAMA also outperforms baselines that learn from unaligned demonstrations (Hotelling, 1936; Ammar al., 2015) by learning maps that preserve MDP structure with more flexible neural network function approximators. For pen+>cart, UMA learns a statemap that outputs out-of-bounds coordinates mainly because the pendulum demonstrations are concentrated around the pole upright state. The optimal UMA embedding matrix in this case is a zero matrix. Then the UMA state map matrix norm is proportional the inverse embedding matrix norm which very large. For snake4 ++ snake3 and reach2 ++ reach3, the MDPs may not be perfectly alignable, yet intuitively share structure. From Figure 4 (Bottom Left) we see that GAMA identically matches two adjacent joint angles of snake4 to the two joint angles of snake3 and the periodicity of the snake's 's wiggle is preserved. On reacher2--reacher3, we find that the central pivot angles are matched and further find correspondences between states that have similar extents of contraction.
7

---

Under review as a conference paper at ICLR 2020
Came Delound
GAYY, (mgroum)
10
50
500
1002
2000
Num Al Aignnentt Tarks
Namt Demondtratton
Dbe Wtaractcee
105
Putformnnt Intormanne Seckkec)
10
50
200
0.0
DvB
1.0
3.2
50mm Alyyneen Tadle
Num Dencenttratinn
Fii Internttions
100
Figure CDIL performance. Alignment complexity (Left), Adaptation complexity (Middle), and transferability (Right) for W2C/R2W on the top/bottom rows, respectively. GAMA outperforms baselines all metrics. Notably, adaptation complexity GAMA is close to that of the self-demo baseline. Error bars/regions show the standard deviation over 5 runs.
6.2 CDIL PERFORMANCE
Wall2Corner (W2C): The self domain reacher2 and the expert domain is reacher3. We use the robot's internal state, action representation. The aligmment tasks are reaching for 12 goals near the room wall centers and the target tasks are reaching for 12 new goals at the room corners, maximally away from the wall goals. The significant difference between training and test goals makes generalization challenging.
The self domain is reacher2 and the expert domain is reacher2-ig tp that has "third person" state space with a 180 camera angle offset. We use the robot's internal state, action representation. The alignment tasks are reaching for goals and the transfer task is writing letters as fast as possible. The transfer task differs from the alignment tasks in two key aspects: the end effector must draw a straight line from a letter's vertex to vertex and not slow down at the vertices in order to trace the letters fast.
Reach2Write (R2W):
Aligmment complexity shown in Figure (Left). GAMA able to learn alignments that enable zero-shot imitation on the target task, showing clear gains over simple pretraining procedure the self domain MDPs in the alignment task set. Other baselines require an additional RL step and cannot achieve zero-shot imitation. Figure 5 (Middle) shows the adaptation complexity. Notably. GAMA-DE DD (blue, dashed) produces adapted demonstrations of similar usefulness as self demonstrations (olive green). Other baselines fail to learn useful alignments from unpaired, unaligned demonstrations and result fails at CDIL. Finally, Figure (Right) shows that the alignment step is useful as weight initialization to accelerate learning of the target task. GAMA (blue) attains optimal performance around 7x faster than all baselines in the W2C experiment, while immediately attaining optimal performance on the R2W task. Baselines fail to learn the writing task an inaccurate proxy reward function harms performance.
6.3 CDIL WITH VISUAL INPUTS
The non-visual environment experiments in the previous section demonstrate the limitations of the time-alignment assumptions made in prior work without confounding variables such as the difficulty optimization in high-dimensional space. In this section, we also demonstrate that GAMA scales to higher dimensional, visual environments with 64 64 x image inputs on the W2C and R2W experiments. Specifically, we train deep spatial autoencoder on the alignment task set to learn an encoder with the architecture from Levine et al. (2016), then apply GAMA on the (learned) latent space. Comparing the dark blue (image input) and light blue curves (internal state input) in Figure 5, we see that the adaptation complexity and alignment complexity of GAMA-DA-ime GAMA-PA-ims are both similar that of GAMA-DA, GAMA-PA and better than baselines trained with the robot's internal representation.
7 DISCUSSION AND FUTURE WORK
We've formalized Cross Domain Imitation Learning which encompasses prior work in transfer learning across embodiment (Gupta al., 2017) and viewpoint differences (Stadie al., 2017; Liu et al., 2018) along with practical algorithm that can be applied to both scenarios. We now point out directions future

---

Under review as a conference paper at ICLR 2020
work. Our MDP alignability theory is first step towards formalizing possible shared structures that enable cross domain imitation. While we ve shown that GAMA empirically works well even when MDPs not perfectly alignable, upcoming works may explore relaxing the conditions for MDP alignability to develop a theory that covers an even wider range of real world MDPs. Future works may also try applying GAMA in the imitation from observations scenario, i.e actions not available, by aligning observations with GAMA and applying methods from Sermanet et al. (2018); Liu et al. 200 Finally, we hope to future works develop principled ways design minimal alignment task set, which analogous to designing minimal training set for supervised learning.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Haitham Bou Ammar, Eric Eaton, Paul Ruvolo, and Matthew E. Taylor. An automated measure of mdp similarity for transfer in reinforcement learning. 2014.
Haitham Bou Ammar, Eric Eaton, Paul Ruvolo, and Matthew E. Taylor. Unsupervised cross-domain transfer policy gradient reinforcement learning via manifold alignment. 2015.
Patrick Billingsley. Convergence of probability measures. 1968.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Zaremba Wojciech. Openai gym. arXiv preprint arXiv:1606.01544 2016.
Norman Ferns, Prakash Panangaden, and Doina Precup. Metrics for finite markov decision processes. In UAI, 2004.
Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, and Pieter Abbeel. Deep spatial autoencoders for visuomotor learning. International Conference on Robotics and Automation, 2015.
Ian Goodfellow. Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farles Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances neural information 2014.
processing systems, pp.
Abhishek Gupta, Coline Devin, Yu Xuan Liu, Pieter Abbeel, and Sergey Levine. Learning invariant feature spaces to transfer skills with reinforcement learning. International Conference Learning Representations, 2017.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, pp. 1565-4573, 2016.
Harold Hotelling. Relations between two sets of variates. Biomelrika, 28, 1936.
Susan Jones. The development of imitation in infancy. Philos Trans Soc Lond B Biol Sci., 364: 2325-2335, 2009.
Girish Joshi and Girish Chowdhary. Cross-domain transfer in reinforcement learning using target apprentice. 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, December 2014.
Yasuo Kuniyoshi and Hirochika Inoue. Qualitative recognition of ongoing human action sequences. International Joint Conference on Artificial Intelligence, 1993.
Yasuo Kuniyoshi, Masayuki Inaba, and Hirochika Inoue. Learning by watching: Extracting reusable task knowledge from visual observation of human performance. IEEE Trans. Robot. Autom., 10:799-822, 1994.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-1333, 2016.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509, 02971, 2015.
YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation: Learning to imitate behaviors from raw video via context translation. arXiv preprint arXiv:/707.033777 2018.
Peter J. Marshall and Andrew N. Meltzoff. Body maps in the infant brain. Trends Cogn Sci., 19:499-505, 2015.
Meinard Muller. Dynamic time warping. Information retrieval for music and motion, pp. 69-84, 2007.
Ronald Ortner. Combinations and mixtures of optimal policies in unichain markov decision processes are optimal, arXiv preprint 19
10

---

Under review as a conference paper at ICLR 2020
Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural computation, 3(1):88 97, 1991. ISSN 0899-7667.
Balaraman Ravindran and Andrew G. Barto. Model minimization in hierarchical reinforcement learning. In SARA, 2002.
Giacomo Rizzolatti and Laila Craighero. The mirror neuron system. Annual Review of Neuroscience, 169-192, 2004.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, and Sergey Levine. Time-contrastive networks: Self-supervised learning from video. arXiv preprint arXiv:1704.0666 2018.
Bradly Stadie, Pieter Abbeel, and Ilya Sutskever. Third person imitation learning. In ICLR, 2017.
Umar Syed, Michael Bowling, and Robert Schapire. Apprenticeship learning using linear programming. In Proceedings of the 25th international conference on Machine learning, pp. 1032-1083 ACM, July 2008. ISBN 9781605582054, doi: 10.1 145/1390156, 1390286..
Tingwu Wang, Renjie Liao, Jimmy Ba, and Sanja Fidler. NerveNet: Learning structured policy with graph neural networks. International Conference Learning Representations, 2018, 2018.

---

Under review as a conference paper at ICLR 2020
Cross Domain Imitation Learning- Supplementary Materials
A HIGH-LEVEL COMPARISON TO BASELINES
Table 2: Comparison of baselines by attributes demonstrated in the paper. The "No Act" column denotes whether or not the demonstrations need contain actions,
Unpaiz Unalign. Data
Zerushox Imit.
Embod. Mismaaihh
Viewpoin Mismarch
Single domann Demo
Act
1PIL (Sinde et al., 2017)
IF (Gipta al., 2017)
L00 al., 2013)
TCN (Semmunnt el al. 2013)
X
X
GAMA (ours)
We note that methods such as IF has potential to be applied to the viewpoint mismatch problem and IfO, TCN have the potential to be applied to the embodiment mismatch problem, albeit they were not shown in the paper. TCN has shown interesting mappings between humans and robots can be learned. However they haven't shown that robots can use these mappings to learn from human demonstrations. Below we summarize the key differences between GAMA and the main baselines.
1. We propose unsupervised MDP alignment algorithm (GAMA) capable of learning state correspon- dences from unpuired, unaligned demonstrations while Gupta et al. (2017); Liu al. (2018); Sermanet et al. (2018) obtain these correspondences from paired, time-aligned trajectories. Our demonstrations have varying length (up to 2x difference) and diverse starting positions. Since good observation correspon- dences are prerequisites to the success of Gupta al. (2017); Liu et al. (2018); Sermanet et al. (2018), our work provide the missing ingredient. Future work could try learning alignments with GAMA, then apply methods from Gupta et al. (2017); Liu et al. (2018); Sermanet et al. (2018) perform CDIL when action information is unavailable from demonstrations.
2. We remove the need for an expensive RL procedure on new target task, by leveraging action information for gero-shot imitation. By learning composite self policy with both state and action maps, we obtain a near-optimal self policy on new tasks without any environment interactions while prior approaches (Gupta al., 2017; Liu al., 2018; Sermanet al., 2018) require an additional RL step that involves self domain environment interactions.
3. We use single algorithm to address both the viewpoint and embodiment mismatch which have previously been dealt with different solutions.
B GAMA MODEL ARCHITECTURE
We now describe the model architecture. function po and discriminators {Dog I are neural networks with hidden layers of size (200, 200).
The state, action map 90 inverse state map fo 1 transition
The fitted policies 1 for GAMA-PA and मr, for GAMA-DA all have hidden layers of size (300, 200). All models are trained with Adam optimizers (Kingma & Ba, 2014) using decay rates 31 = 0.9, 32 = 0.999. For the spatial autoencoders used in GAMA-PA-img and GAMA-DA-img, we use the same architecture as in Finn al. (2015) We a learning rate of le-4 for the alignment maps and 1e-5 for all other components. These parameters are fixed across all experiments.
C BASELINE IMPLEMENTATION DETAILS
In this section we describe our implementation details of the baselines.
Obtaining State Correspondences We use 5000 sampled trajectories in both expert and self domains to learn the state map for IF and CCA. For UMA, we use 20 sampled trajectories to learn that in pendulum and cartpole environment and 50 trajectories in reacher, reacher-tp environment (much beyond these numbers UMA is computationally intractable). For IF and IfO, we use Dynamic Time Warping (DTW) (Muller, 2007) to obtain state correspondences. For IF, DTW uses the (learned) feature space metric space estimate the domain correspondences. For IfO, DTTW applied on the state space. We follow the implementation procedure in Gupta et al. (2017).
To visualize and quantitatively evaluate the statemaps learned prior work, we compose the encoder and decoder for IF and use the Moore-Penrose pseudo inverse of the embedding matrix for UMA and CCA.
12

---

Under review as a conference paper at ICLR 2020
Transfer Learning In the transfer learning phase for CCA, UMA, IF, and IfO they define a proxy reward function on the target task by using the state correspondence map.
T Praxy (s
E (lf(()))) -(s5"))
1 IT
11
(t) is the where (+) is a self domain state at time I, T is the collection of expert demonstrations, and 8y.7 expert domain state time in trajectory т. IfO additionally defines penalty reward for deviating from states encountered during training. We refer readers to their paper (Liu et al., 2018) for further details. The transferability results of Figure 5 (Right) show the learning curve for training on the ground truth reward for the target task where the policy is pretrained with a training procedure on the proxy reward. All RL steps are performed using the Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al., 2015) algorithm.
Architecture For UMA and CCA, the embedding dimension is the minimum state dimension between the expert and self domains. For UMA, we use one state sample every 5 timesteps to reduce the computational time, and we match the pairwise distance matrix of 3-nearest neighbors.
For IF, we use 2 hidden layer with 64 hidden units each and leaky ReLU non-linearities to parameterize embedding function and decoders, the dimension of common feature space is set to be 64, The optimizer are same with respect to our models and the learning rate is 1e-3.
For IfO, we use the same architecture as the statemap in GAMA for their observation conditioned statemap.
For TPIL, we use 3 hidden layer with 128 hidden units each and ReLU non-linearities to parameterize the feature extractor, classifier and domain classifier. We use Adam Optimizer with default decay rates and learning rate 1e-3 to train the discriminator and use same optimizer and learning rate with respect to our model to train the policy.
D ENVIRONMENTS
We use the 'Pendulum-vO" 'Cartpole-vO' environments for the pendulum and cartpole tasks which have state space (w, u)) and (w. th, x. i)), respectively, where w is the angle of the pendulum/pole and x is the position of the cart. The action spaces are (F.,) and (Fx) where F,, is the torque applied to the pendulum's 's pivot and F, is the x-direction force applied to the cart. For snake3, snake4 we use extension Wang al. (2018) of the 'Swimmer-vO" environment from Gym Brockman et al. (2016). A K link snake has a state representation (W1, ..... WK, W1.... w'k where W; is the angle of the ht snake joint. The action vector has the form reacher environments were extended from the 'Reacher-vO' gym environment. A l link reacher has state vector of the form (w1..., WK, th,,... W'K, Zg. Ug) where wi is the angle of the 1oth reacher joint and (xgg Mg) is the position of the goal. Note the key difference with the original Reacher-vO environment is that the difference vector between the end effector and the goal coordinate was removed from the state to make the task more challenging. The action vector has the form (w/ where Fu is the torque applied to joint k. The third person reacher environment (reach-tp) has an angle off set of such that (w) + π. ..., WK + T, W1, ..., W'K, g. 3g) corresponds to the same internal robot state as (wi,.., WK; ti,,..., ..., wic Fg: Mg) in the original reacher domain. Note that the goal coordinates (Igg Mg) have not changed but the joint angles have changed. Thus proper alignment between the original domain and the third person domain should add an offset of to the original state. For the writing task, the vertices of the letters to write are specified sequentially with the goal coordinates. Once the first vertex is reached, the goal coordinates are updated to be the next vertex coordinates. The reward function is defind as follows:
Fux) where Fwe is the torque applied to joint k. All
Rueitte (8) = 100 if state s corresponds to reaching vertex
else
Thus the agent must perform a sequential reaching task and accomplish it as fast as possible. The key difference with a normal reaching task that the reacher must not slow down at each vertex and plan it's path accordingly in order to minimize drastic direction changes. Further more the reward is significantly more sparse than the original reacher reward which gets reward inversely proportional to the distance between the end effector and the goal.
13

---

Under review as a conference paper at ICLR 2020
E VIDEOS OF ALIGNMENT MAPS
We will provide real-time visualizations of the learned alignments in the final version of the submission.
F PROOFS
Definition 1. An optimality function OM, S₂ x Ar + {0,1} for an MDP Ma satisfies:
OM, (82: (z)) = 1 3 E M,,, such that (82, az) E supp((n;) and O., (8xx az) = otherwise.
Definition 6. An optimal policy na is covering if 0, (82, Ux) -1 az Supp(TT
We first restate the definition of MDP reductions:
Definition 2. An MDP reduction from MM = (S, An. Ba:
Bas Ra) to My = (Sy. Ay. Ny,
Fu Rgy) is
tuple (tb 15) where & Sz + +Sy 1b As + Ay ure maps that preserve:
1. (optimal policy) 48z E Es,,0z E An,8y 8y E Sy. Gy E Ay.
Om, (0(s2), (ax))) =1 = (8,, Ax) =1 OM (sy, ay) =1
(1)
(-y + (,b5((a)) +
(2)
2. (dynamics) Vays byy E Sy ay E A, where OM,, (sy; ay) =1,
Py = (PP((32: az)) Vsg E x "sy()), az E 5/ (ay)
(3)
where we define (5y) (0xx)( sy), v (ay) (axlv(ax) ay)- Furthermore, is MDP permutation if and only if o, v0 are bijective maps.
Definition 7. MDP Ma is unichain, all policies induce irreducible Markov Chains and all stochastic optimal policies induce ergodic, ie irreducible and aperiodic, Markov Chains.
Assumption 1. All considered MDPs are unichain with discrete state, action spaces and deterministic dynamics i.e. P: : S.Sx x A + S. Furthermore, there exists dummy state, actions s a where OM,ss, a²) 0 Vs& E S and (s a.) = 0 Va E A
As stated in Assumption 1, we specifically consider MDPs that are unichain with deterministic dynamics. We note this is reasonable assumption since physics is largely deterministic and many control behaviors, such as walking, are described by unichains. We now formalize a notion of MDP alignability starting from defining class of structure preserving maps between MDPs. We first prove some lemmas that will assist in proving the main Theorem. Recall that f S. - Sy Ay + As and के. = =g By f
Lemma 1. Let My,My be MDPs satisfying Assumption I (see Supp. Materials), Mr Zo.v My and Ty be optimal in My- Vg: : Ay + Ar s.t vp g(ay) = By Way E {aylEs, E Sy s.t O.,, (Sy. ag) 1}. it holds that क = g Ny o @ is optimal in M..
Proof. Without loss of generality, consider an arbitrarily chosen sample a, = g(ay), ag ~ (.((((()))) for any 8g E Sz. We first see that:
Om, (ō(sz); v (ax)) = OM, (#(s=), (((()) = (0(2))
=1
(9)
where the first step substitutes 0g = g(ay), the second step applies (bog(ay) = ay since ( (z2) ag) =1 due the optimality of Ty. and the last step follows from Corollary 1. Since (ò, e) is a reduction, we have that O,, (ó(sz), (aa)) = + OM, (8z, az) = 1 by Equation (1). Therefore, OM, (82, ax) = 1 Vs, Sz, Va, (upp(#z(() (-ls.))) Then by Lemma 2, मं is optimal.
Lemma 2. Let MDP M, satisfy Assumption 1 and T (ag(s,) be (stochastic) mixture policy that chooses 02 randomly from
ax) 1). Then, na is optimal. (Ortner, 2005) Corollary 1. Let MDP M2 satisfy Assumption 1 and Tu be optimal. Then O,, (s2, 1 Vs e Sz, ar E supp(T=x(11sx))
Lemma 3. Let MDP M2 satisfy Assumption 1 and Te be stochastic optimal policy. Then the triplet
stationary distribution pT_ (8z, 0z (yy) lim Pr(s 8z. oz (1) exists and is unique.
(1+1) 8'y; uay Pa. mn))
Gzz 85
14

---

Under review as a conference paper at ICLR 2020
Proof.
PR. (8z) Oz, (/) = lim Pr(s = 8R, (t) = az 82
(1+1)
s Twy Pai Tm.)
t1:00
lim Pr(s S2i Tay Pa. N2 Th (az 15 16s = Pa az))
= (= 11 Pa az))
lim 1000
Pr(s (1) = Szi Tz Pay 7ly
where 1 is the indicator function. The limit in the last line is the stationary distribution over states, which exists and is unique since stochastic optimal policy induces an ergodic Markov Chain over states.
Lemma 4, If a real sequence { converges to some a IR, then
lim ΣJ a lim ag ==a T T
Proof. Denote Ar = E/= di and Br T. We have
Ar+1 Ar BT By
lim T-00
lim 4T+1 = T-000
(10)
According to the Stolz-Cesàro theorem,
AT+1 Ar
lim AT T-000 BT
lim T BT+1 Br
the limit on the left hand side exists. Therefore
AT lim Ei= = =a T T
lim T Br
(11)
which completes the proof.
Recall that our target distribution o4, and proxy distribution oz .yy
were defined as:
04 ay: ay: ())) = lim
1
Σr- Pr(s)) (t) = Sy, ay ay.
(t+1)
s Ty. Py. nv)
(12)
T00 T
x R2
(s,, (y) = lim
1 -T
Pr(s (t)) = Sy ay ay, By (t+11) s'y3 P)
(13)
T T
We are now ready prove that our proxy and target limiting distributions exist. Lemma 5. Let MDP My satisfy Assumption I and Ty be stocahstic optimal policy, Then, y (Syn Ugy k/) = ph (8ys Ug 8y))
Proof. Recall that the stationary distribution p4 (sys ays syy) is the following limiting distribution:
pa (Sys Og 8')) lim Pr(s)) = sy, a ags o ((+1) 8رy Ty. Pay Thy) f000
(14)
pH, (sy, oy, ( exist for My as shown in Lemma 3. Then,
(8y, ay: (/))
lim ΣT- Pr(s (1) sy, Ug (1) ag 8y
((+1) 8ky5 Tyy Per Py)
(15)
T-x00 T
lim Pr(s (t) = Sy, a agg ((( ((+1) s'ys Tg. Py. (y)
(16)
P", (Sy: Ag. 5,)
(17)
as desired. The second line follows from Lemma 4 and the last line follows from Lemma 3.
Lemma 6. Let MDP My satisfy Assumption and ny be stochastic optimal policy. Then, supp(o".) ((5g) ay (,,10 M, (sy, ar) 1, L, S/ E Sy. ag E Ay
15

---

Under review as a conference paper at ICLR 2020
Proof. Assume for contradiction that there exists (sy, 5/) supp(o ) but (sy: 8gg) &
(88y) Byz M, (8y:ay) Sy. Ky Eyy By E Ay). Then OM,, (8y, Og) 0. Since
0 (5ys uy ay) lim Pr(s Sy, a/ oy 8y s((11) = ='y; Ty. Py. Bly)
Ty
lim Pr(s)) = 8,) Pr(ay (1) = ayls = 8yy) Pr(
Pr(ss (1+1) = 85y5 (1) =8y, ny = an)
(-20
lim Pr(s)) sy) sy) (8y) Pr(sf(+1) 5yy1% 8y, a4) 0gy
1-:00
0
[3) Pr(s s.ylo Sy: d an) lim Pr(s syy)
400
First line follows from Lemma 5 and terms taken out of the limit in the second last line since the stationary distribution over states exist as Mg is unichain and Ty is stochastic optimal. Ty ( (sy) 0 since O,,, ay) (y sy) from Corollary 1. Then, we have 72 (sy: Oy. ay) which contradicts (8y: By: kyy) E supp(aX,) concluding the proof.
16

---

Under review as a conference paper at ICLR 2020
Lemma 7. Let MDP My satisfy Assumption I and Rz 9 o ny f be stochastic optimal policy in M2 where f S + Sy is the state map, g : Ay - Ag is injective action map, and Ty is a stochastic optimal policy in My Further let F Sz x g(A,) S₂ Sy x Ay Sy be the map
= (f(a),, f()) Then, 0=a_ (sy Oy, 8/)) Flp (52, Oz2 8z)).
(0) (1) = for t = 0,1, ... where Proof. We first define the triplet random variables X (s 0g
(t) (t+1) for t = 0, 2, were defined in Definition F is function on supp(p",) ) E S₂ x g(A,) x S2 and F(X(1)) = (s!)) ag s Furthermore, since F is a function defined on a discrete domain and codomain, there always exists trivial continuous extension of F. We may thus apply the continuous mapping theorem (Billingsley, 1968):
So au 82
x 4 x F(X(0) $ F(X)
Since Ma is unichain and #t is stochastic optimal, the distribution of (() converges (in distribution) to Ph (Sx, ags 8/2)) as + oc by Lemma 3. Applying the continuous mapping theorem, it follows that
the distribution of F(X()) (5 By By 1) converges (in distribution) the pushforward measure
(4)
F(p%, (8x, az, s())) as t + 0D
Directly applying this result, we obtain:
x (sy, Cy syy) = lim
T-000 T El= Pr(sy) (1) = Sy. (t) (t+11
1
oy. 8y
syy; P)
(18)
= lim Pr( = sy. ab) = yy y
g(t+1) و P)
(19)
10000
= =F(Ak, (82,0z, 8,))
Oz
(20)
as desired. Line (18) + (19) follows from Lemma and (19) - (20) follows from the continuous mapping theorem.
Lemma 8. Let X,Y be countable sets, o oX - Y be function, and I be the indicator function. We denote o (y)) = (x](((x) = y}. Then Vx E xx Y
1 = x()) = -xco-1() 1(x=2)
Proof. Since both the left and right hand-side of the desired equality only take on values in {0, 1), it suffices show the following statements hold for arbitrarily chosen X, Y:
Exeo (()) 1(x 2)
1(y d(x)) =1
1 = x(x)) = 1 + E.c0-10) 1(x =) =1
For the first direction, we see that 12c0-100 (v) 1(x :) 1, then x E - and thus (xx) y.
For the second direction if 11y = (xx) = 1, then x E ()) Thus there exists unique 2 such that and E o (y)) Then, Σ Exeo ((x) 1 = 1 as desired, which concludes the proof. U
17

---

Under review as a conference paper at ICLR 2020
Finally, we prove the main theorem. Our objectives are:
1. a is optimal
2. a= ozH
3. g injective
Theorem 1. Let My be MDPs satisfying Assumption I (see Supp Materials). If M2 MMy then : + S. 9 Ag + An. and an optimal covering policy Ty (Supp Materials, Def 6) that satisfy objectives 1, 2. Conversely, if S. Sy,9 Ay Ar and optimal covering policy my satisfying objectives 1, 2, 3, then Mr 2 My and ( 1) E (MM,, My) S. f = @ and th o g(ay) = ay Vay E Ay-
Proof. We first show the (+) direction. Using any (Ø, v) E r(M Mg) we construct f and g in the following manner: f(sz) ((x2) vs, E Sg. g(ay) maps to an arbitrary chosen element from the set vi (ay) = {az (ax)) = dy) if & v A 9 and an arbitrarily chosen action ag Ar otherwise. We see that wa, Ag for which Esy Sy such that O.,, (sy: ag) 1. it holds that ( # 0 by Eq 2. Therefore, 1è o g(ay) = ay Vay E Ag for which Fsy such that OM,, (8y, By) = since 1b maps all elements in (ay) to ay. For Ty we choose any covering optimal policy for My. It suffices to show that this choice of J, 9, Ty satisfies objectives 2.
Objective 1. Ω. is optimal: follows from Lemma 1.
Objective 2. a 0 Since 6 is reduction, it follows that Vs, E Sy: By E Ay such that (sg, ay) 1, any s'y E Sy. and Vt 0, 2....
Pr(af = 1511850 = sy, o.))
Σ Pr(aftt1) = =5115 1) 1500 a 0,) Pr(e! =6.18/) =y. oz)
-es.
Σ Pr(a)t1)) = =/1611 x()
Ke8,
Σ Pr(s/(+1) 1/11600 o0 50) a/ a,)
a ag1s5 a/. ax) Pr(s
Σ I(s, = =((())
Σ Pr(s((t1) 4/10000 = Pr(s =xlal0 2.. ag. Pr ag1s!! =2. 2.)
Σ 116 = =(()))
Σ 1 = Pa (...
= = e.) =a.lah =ay)
Pr(aft = =,yl = Pr
Σ 11s = ((())
Σ
16 = a.()
Pi(af = sylos + Pr(e! Ila = o(a,])
6,E..
11,, = a( (r()) =
Σ
16 ((2))
Σ
11 P. 9(a,)))
4
Σ 1(5y = 2( P
JE5.
Pr(s/ =x,)
Σ
Σ 161 P (s...(()))))
(1g)) 6,0)),0001 (s,)
Σ Pr(s = x
Pr(et 10 =zz)
Σ
1 = p.(,s)((a,)))
K.86 (p,))
Pr(e! = ((())n (
Pr( (0) 8x)
Lemnu
Σ
11 1(5, of P. (c..))))))
NED ((v)
Σ
!1( = x()
Eg8
Σ
ax)
1 a,))
2,60-115 (bg))
MeE (ts)
1(6/ = PP((9;
Pr(sf(+1= K,16(00 = Ky, a.00 = 0z)
(21)
Furthermore, from Definition 5, we have:
Pr(af = ay (8(0) = sy) = (y((y18)) = Pr(aa)) = ay) = 8y)
(22)
18

---

Under review as a conference paper at ICLR 2020
Then, Vsy. Syy E Sy and Vt = 0.1.
Pr((((+1= syl$ =sy)
Σ
Pr(sl(t)) =8yb8 8y, agp (ly) Pr(ay) (1) aayl8 88)
ayEd,
Σ
Pr(*g (1+1)
Byl sylo8 By: a2 = aly ((
o, a.Esupp(T (11x,)
Pr((fttt) 8y (y) = 8y)
(23)
we are justified in the substitution for the dynamics in the second line since OM, (sy, kyy 145, E Sy: By E supp(Ty(-(sy)) by Corollary 1. Since My is unichain and ny is a stochastic optimal policy, the stationary distribution lim Pr(s (t) sy) is invariant to the initial state distribution f/y and only
depends on the state transition dynamics Pr(s (1+1) syyls (0) 5y)) Equivalently any stochastic process with the same state transition dynamics will converge to the same stationary distribution regardless of the initial state distribution. Thus,
lim Pr($ = syy)= = lim Pr(s = syy Vsy E Sg 6000
(24)
Finally putting these results together, the following equalities hold for (8y, Byy 86/) (00y ayy M, (8y. ay) = 1, Ky: Ky E ay E Any
E
Rz. a (8y, ayy 2/y)
Lemma
lim Pr(A Sy: a( ay, 8/ g(t+1) sy; 1
lim Pr(sf syy pr(aa) ayl (/1 Syy Pr( 5 sy. a/) =ay)
1+000
(2))2)
lim Pr( = sy) Pr(aa) (/) syy Pr(s (((((t))) $/11880) Sy. al) = ay)
100
- M[5, (s an)) lim Pr(s sy)
EQ(24)
(y(a 88 [6( = (s (y)) lim Pr(s 8n)
lim Pr(s = sy) syy Pr(aa)) (t) agy ayls!!) sy) Pr(s = s7,18h4) = sy. a. = ay) 10000
Lemma 0 (8y, By: syy
The constant terms moved in and out of the limit in the fourth and sixth line since the stationary distribution over states exist as M, is unichain and Ty is optimal for My. This allows us to conclude that o -- o% since o4 is supported on ((5g. By 1,,,, (8y, Qy) 1, &y E Sy: ay Ay) by Lemma 6.
19

---

Under review as a conference paper at ICLR 2020
Now show the (=) direction. We first introduce some overloaded notation:
o (8x) lim
1 T-1 Pr(s Szi ueg Px: no))
-t=
Lemman lim Pr(se (9)
Sui 22 Pas 770)
T-000 T
ok (8a, (z)) an)
lim
1
-1=0 Pr(ss
2-1
(1)
Say (1) Ozi Hay Pa 1a)
T+x T
Lemmna 5
Szz a Uzi uzy Pz: 7s)) lim Pr(s (1(
10000
lim Pr(s! (1) Saj ue Pas 122 ar (Ox(2)))) 1-00
( (x)
lim Pr(s 8x; uz, Pa, 11)
az (x) (5x)
(25)
Then,
fig (Sx, az (//z)
Lemma 5
lim Pr(s(() = 82, (1) = az. 5/ = 8/ $hitt ug Pa.
(1+1)
10000
lim Pr(s(t+1)_ = 5'115 = 3zz az = a,) Pr(s! = 8xx a = azi fz; Pay 7x))
+x
1(s', = P2(521 ax)) lim Pr(s (1) = Sz, a = axi M2z 7z)
10x
1 = (8 Og) )z (82,
(26)
1(s/ PP az) (1= (8x)
(27)
Given f. that satisfy objective 1, 2, and 3 we construct (ø) vb as follows and show that (d) 1)) E
I(M,, My):
[8x)
8f E supp(at, (sx))
o(xx)
sd y/
otherwise
((x)) gg-1(a>) if As. Us,es, (upp(
otherwise
where sd a are dummy state, actions such that OM, (sy. ay) = Vay E Ay and OM,, (sy, ag) sys O Vssy Sy- Such dummy state, actions always exist per Assumption 1. Mapping to the dummy state, action will ensure that the constructions will not map suboptimal state, action pairs from domain æ to optimal state action pairs in domain y. The following statement holds for our construction (ø, v):
(8 a E supp(a", (8z, (z)) +
OM, (M,(6(5))) (6()), = Vs E Sny dg E A
(28)
We first prove the forward direction: (s ax) E supp(o (52, ax)) fa of, (sn)aa(a" (a]ls)) > 0, so oi (st) > 0, ie St E supp(a (sx)), and Ít. (a])))))) > 0. Furthermore,
Th (s),a) Eq25
it (a > 0 = (az) (upp(r,((([((t))) since is injective. To see this, assume 3(5 a%) such that (() > 0 but g (at) ) & supp( (-I.(((()((s Then there must exists df E
such that a', (a)) but g(ay,) ((o)))=(=-(())) (a))) = o contradicting the injectivity of g Ay. Putting these results together we obtain ((aa)) = (a)) supp(Ty(
(1)(((((())))))) ()) Since My is stochastic optimal policy and M, is unichain, v(at)) E supp(m,(-lo(s:))) + OM, (o(s)), ((a))) = by Corollary 1.
For the converse direction we prove the contrapostive: (s) & supp(o (82,az)) ax)) (M,()())
O Var Sr, Ar We exhaustively consider all cases in which (s7,0)) & supp(až, (82,0z)) ie of (8%,at) F9g5 *n(atlst)) (st) = 0. If o (8%) = 0, then of & supplo (8z)), so DM,(0(s)) ay) = (M, =0 Vay E Ay- Else if (a(l)))) = 0, of (st) > 0 and an $ As then (s,
= -M,, ad) 0 Vs, E Sy. Finally, consider the case # (agls)) = 0, (st) > 0 and at E Ar Assume for contradiction that OM, (d()),, (aa)) = 1. Then, w((a)) E supp(T(((1({{)) since my is covering optimal policy from Definition 6, which implies g (aa))) E supp( (s+)) since of (st) > 0 and o E Aa.- It
20

---

Under review as a conference paper at ICLR 2020
follows that (at)) E supp(o(T,((1((:)))) a supp(#a((1s$) since (xx()s)) is the pushfor-
ward measure (-f(()))()))) Then, oh (5 at) Eugs oi ( > O, since Rz (= ss)) > 0 and oi ($t) > 0, which contradicts ($ a.)) supp(o (s., az)). This concludes the proof of Equation 28.
We proceed to show that the optimal policy and dynamics preservation properties hold for our construction (ò, e
Optimal Policy (Equation 1): From the converse direction of the above subelaim and the optimality of tx the result immediate follows:
OM,, O,,, (M,((())) =1 Fg28
(5 az)) supp(o (x))
Eg25
n (a >0
Coel
OM. ag) = 1 Vs E Sz, az E A2
Surjection (Equation 2): Assume for contradiction {(s any such that Om, (sy, an) = 1, but Φ (5,) 0 (a 0. Since O,,, (sy, (z) 1 we have sy A ag:az & az. Thus o(5;)) -1 (5,) {o(a;))) Since g is function defined Vay E Ay, follows that (an) + 01. Thus it must be that Φ (s5) = 0. Let sy = Pa Gy) Then, *,
and si- (07) = {(g (
x-y (sy, ay s
Lemma
10000 q sy ay agy 1((+1)) sT))
lim Pr(
lim Pr(s((+1)
sy | sy 600) = at) Pr(s sy a.) ag)
1-000
9
lim Pr(a((+1)
s </1%// s a o.) Pr(af) ag1l(h) = 84) Pr(S =8)) sy 1116 by a d n (51)) E,, Es Pr(se (1)
1-000
lim Pr(s((4J)=
sx)
t-xx
lim Pr(sf(+1)= s ( sy 1 = ())m t-000
0
However,
0²
(8g By s
Bq27 1 (s o.) = =P (aglo)))))) ($)
= T Jo ($ > 0
To see why the last inequality holds, first recall that My is unichain and Ty is stochastic optimal for Mg. so the stationary distribution over states have full support over S, (: stationary distributions of
irreducible markov chains are fully supported over the entire state space) Therefore 0f (5g)
Lemmañ
lim Pr(s (1) Sy: Ty. >a)>0 O Vsy Sy. Thus, we have 24 (s)) > 0. Furthermore, Ty T (1j)s)) 0 by Corollary 1. Putting these two results together, we obtain xy (sy) ay st > 0. Then, 0 #2 + 0² which contradicts the satisfiability of objective 3.
Dynamics (Equation 3): Assume for contradiction that Bs, a and s = Po af such that OM, (0(s7),) = 1 but the dynamics preservation property is violated, ie Py(o(s ) () # ((Pa(s = o(s If (s_, & supp(o (s2, az)). then OM, (a())). = O Equation 28 which contradicts ( ), ((a7)) = 1. Thus, it must be that (s, .,)) E supp(of (s., az)) which further implies (sH ar s E supp(o (S2, az, s())) by Equation 26 and x(s;) = f(5;) ) (aa()))) = g (a7) by Equation 25 since 0f (x5) > 0, (a >0
Fet:FF S. x g(An)) X Sn + Sy x A, x S, be a function (a, b, + (() Then, by Lemma 7, we have oF. - (Szy ary { Flp R. (8z, az. az, {y,)) Lommas
Floz t (8z, Oz, 8y)) So,
t O (3 ; 3/5 0 xx + [F( 55 , = tF "1'5 1( > 0
21

---

Under review as a conference paper at ICLR 2020
Thus, v(( 2 o² v() p((o(s))
(0(s7), ((5 (82. (82: ax 84)). However,
x
a. Thus, supp(o o supp + b- * o which contradicts f.g satisfying objective 3. This concludes the proof of the main theorem.
22


Reviews:
[{"review_id": "Hyx8KLvE5S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a learning approach for zero-shot imitation learning in an RL setting across domains with different embodiments and viewpoint mismatch. The proposed approach involves two steps, alignment and adaptation. In contrast to previous work, the alignment between domains, represented as MDPs, is learned from unpaired, unaligned samples from both domains. The paper presents a theoretical formulation of the cross-domain imitation learning problem, and presents an algorithm for training alignment and adaptation from data. \n\nI think the paper is well written and theoretically well-founded. The authors provide experiments comparing to many previous works in cross-domain imitation learning and show that their approach outperforms previous approaches. \n\nI only have minor comments.\n\n1. For imitation between reacher2 and reacher3, there are multiple correspondences between the two domains due to the redundancy in the 3-link robot (and in general with n-link robots). How does GAMA deal with these redundancies? For example, an n-link robot is able to perform the same task with different configurations (elbow up, elbow down, etc.,) and all these will correspond to one configuration of a 2-link robot. \n\n2. In practise, can 'alignment complexity' and 'adaptation complexity' help identify if transfer with GAMA between two domains is not beneficial? Results in Figure 5 only shows cases for GAMA in which the two metrics are good, resulting in good transfer. However, I am interested in knowing if there could be cases in which the two complexity metrics tell beforehand that transfer will not be beneficial. \n\n3. In Figure 5's caption, Adaptation complexity is on Left and Alignment Complexity in the Middle. However, the text refers to Alignment complexity on the Left and Adaptation complexity in the Middle.\n\n4. A reference is missing in Appendix B.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJgFzgcaYS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes Generative Adversarial MDP Alignment (GAMA) for imitation learning. Given a set of paired MDPs, GAMA learns a state mapping f and an action mapping g such that one MDP can be reduced to another. For a new test MDP pair (x,y) where expert demonstrations are available for y, GAMA can use f to map a state of x to a state of y, mimic the expert behavior, then use g to map the expert action back to an action in x. The reduction is theoretically motivated, the optimization is based on adversarial learning and finally, experiments on common Gym environments show that GAMA can achieve effective transfer.\n\nPros\n- The writing is great and easy to follow\n- The method is theoretically motivated\n- Experiments prove effective\n\nCons\n- The proposed method may not work well for complicated environments\n\n(1) In the discussion after Def.4, given an alignment task set D_{x,y}, how do we know whether a common (w.r.t. all MDP pairs) reduction exists? In other words, how do we know that the MDP pairs are from the same equivalent class (joint alignable) in practice?\n\n(2) The alignment needs to learn a lot of components: state mapping f, action mapping g, domain dynamics P^x. The domain dynamics can be difficult to learn for complicated environments, which may jeopardize the learning of f and g as a result because they depend on the accuracy of the learned dynamics. The experiment only uses the hidden representation for the image as input. Such lower-dimensional representation is not always available.\n\n(3) Experiment:\n- What happens for the UMA in Fig.4 top-right?\n- Table 1 only has the results of three alignment tasks. How about the rest?\n- What are the error bars in Table 1 (and also Fig.5)? Based on how many runs?\n\nMinors:\n- It is more common to use \"state-action pair\" instead of \"state, action pair\".\n- Sec.6.1, task task exemplify", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "H1g_go22tH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\nThis paper proposes a method to perform alignment between demonstrations in different domains in order to adapt a policy from expert domain to the test domain. They also include a formalism that studies alignments of different MDPs.\n\nStrengths\n1) The paper presents a formal framework to study many recent methods in cross-domain/cross-view imitation learning with a common lens. \n2) The solution proposed by the authors might have a big practical advantage for cross-domain imitation learning settings. They claim that they do not need to learn the policy using RL in the new domain as they learn the action mapping between the two domains to retrieve the corresponding action in new domain based on the action in expert domain. This is a very interesting result.\n3) The solution is quite modular with 3 important parts: i) state mapping network, ii) action mapping network, iii) dynamics model. \n\nWeakness\nThe biggest weakness of the paper is in the evaluation section. \n1) Issues with baselines:\ni) There seems to be some problem with UMA. In Figure 4, it cannot figure out the alignment with self domain. Why is that the case? Where is the cartpole in UMA row for pen-cartpole alignment ? Given such poor performance during alignment is it fair to compare to UMA in the subsequent section?\nii) Why are the other methods not able to align well when there is no domain shift? \niii) How is dynamic time warping actually used to calculate the state correspondences for training IfO and IF? Is it calculated on top of the states?\niv) How are the policies trained for the transferability task? Which RL algorithm? \"Baselines fail to learn the writing task as an inaccurate proxy reward function harms performance.\" \n2 aspects are involved here:\n1) initializing from alignment network\n2) using rewards from the state representation to train the policy.\nDo the algorithms still fail if the true reward is used with the alignment initialization? \n\nReward scaling is known to be important for RL algorithms[1]. May be features from these algorithms have to be scaled to scale the rewards for the RL algorithms to work.\n\n2) Issues with environments:\ni) The environments considered for domain transfer seem quite trivial for the alignment between them to be a big problem. For example, the authors present the task R2W: reacher2-tp that has a \"third person\" state space with a 180 camera angle offset. I am not sure how this rotation changes the state for the policy that uses states as input. One of the states of reacher is vector between goal and state which should still be informative enough to solve the task. Presumably the rotation creates a different view for the image based experiments. There also the transformation is quite easy for the spatial autoencoder given that it has access to coordinates through the spatial softmax layer. \n\nii) It is unclear what the R2W task entails or how difficult it is. \n\n\"The transfer task is writing letters as fast as possible. The transfer task differs from the alignment tasks in two key aspects: the end effector must draw a straight line from a letter’s vertex to vertex and minimally slow down at the vertices. \"\n\nIt seems there is a term that penalizes the policy from doing things quickly. Some questions regarding this task:\n1) How is this task specified? \n2) Is agent rewarded for slowing down too?  Does the goal of the reacher and the state representation corresponding to the vector between end effector and goal get updated once it reached a particular point? Does the agent have to write one particular letter? Why do the other baselines not work at all on this task?  \n\niii) Is it possible to run experiments on environments presented in Invariant Features paper so that the importance of CDIL can be assessed better.\n\n4) \"HIGH-LEVEL COMPARISON TO BASELINES\" section is misleading. While the different baselines (TCN, IfO, TPIL) were developed in the context of view mismatch that does not mean the algorithm cannot be applied for embodiment mismatch. For IF also the same idea can be applied for viewpoint mismatch. TCN also proposed single-view version which can be used to learn representations without paired data. Hence, it is not right to say TCN can't work with unpaired alignment data. It also needs to be stated in this section that CDIL requires access to actions while some baselines like TCN do not need actions that can be used for training.  \n\nQuestions\n1) How were the alignment videos generated?\n\nDecision\nThis paper has the potential to be an important paper in this field. But at this point needs further empirical evaluation with stronger baselines and known benchmark environments.\n\nMinor Comments:\n1)  \"boltmzman machine reconstruction error\" - Boltzmann\n \nReferences\n[1] \"Deep Reinforcement Learning That Matters.\" Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, David Meger", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a novel approach for imitation learning in settings where demonstrations are unaligned with the task (e.g., differ in terms of state and action space). The proposed approach consists of alignment and adaptation steps and theoretical insights are provided on whether given MDPs can be aligned. Reviewers were positive about the ideas presented in the paper, and several requests for clarification were well addressed by the authors during the rebuttal phase. Key evaluation issues remained unresolved. In particular, it was unclear to what degree performance differences were purely caused by issues in alignment, and reviewers did not see sufficient evidence to support claims about performance on the full cross domain learning setting.
Decision: Reject