Cross-Domain Imitation Learning via Optimal Transport
Authors: Arnaud Fickinger
Samuel Cohen
Stuart Russell
Brandon Amos
Venue: ICLR 2022 Poster (2022)
Keywords: optimal transportation
imitation learning
cross-domain imitation learning
gromov-Wasserstein

Abstract: Cross-domain imitation learning studies how to leverage expert demonstrations of one agent to train an imitation agent with a different embodiment or morphology. Comparing trajectories and stationary distributions between the expert and imitation agents is challenging because they live on different systems that may not even have the same dimensionality. We propose Gromov-Wasserstein Imitation Learning (GWIL), a method for cross-domain imitation that uses the Gromov-Wasserstein distance to align and compare states between the different spaces of the agents. Our theory formally characterizes the scenarios where GWIL preserves optimality, revealing its possibilities and limitations. We demonstrate the effectiveness of GWIL in non-trivial continuous control domains ranging from simple rigid transformation of the expert domain to arbitrary transformation of the state-action space.

Published as a conference paper at ICLR 2022
CROSS-DOMAIN IMITATION LEARNING VIA OPTIMAL TRANSPORT
Arnaud Fickinger 13+
Samuel Cohen 23 Stuart Russell' Brandon Amos
Berkeley Al Research University College London Facebook Al
ABSTRACT
Cross-domain imitation learning studies how leverage expert demonstrations of one agent to train an imitation agent with a different embodiment or morphology. Comparing trajectories and stationary distributions between the expert and imi- tation agents is challenging because they live on different systems that may not even have the same dimensionality. We propose Gromov- Learning (GWIL), a method for cross-domain imitation that uses the Gromov- Wasserstein distance to align and compare states between the different spaces of the agents. Our theory formally characterizes the scenarios where GWIL pre- serves optimality, revealing its possibilities and limitations. We demonstrate the effectiveness of GWIL in non-trivial continuous control domains ranging from simple rigid transformation of the expert domain to arbitrary transformation of the state-action space.
Imitation
INTRODUCTION
Reinforeement learning (RL) methods have attained impressive results across a number of domains, e.g., Berner et al. (2019); Kober et al. (2013); Levine et al. (2016); Vinyals et al. (2019). However, the effectiveness of current RL method is heavily correlated the quality of the training reward. Yet for many real-world tasks, designing dense and informative rewards require significant engineering effort. To alleviate this effort, imitation learning (IL) proposes to learn directly from expert demon- strations. Most current IL approaches can be applied solely to the simplest setting where the expert and the agent share the same embodiment and transition dynamics that live in the same state and action spaces. In particular, these approaches require expert demonstrations from the agent domain. Therefore, we might reconsider the utility of IL as it seems to only move the problem, from design- ing informative rewards providing expert demonstrations, rather than solving it. However, if relax the constraining setting of current IL methods, then natural imitation scenarios that genuinely alleviate engineering effort appear. Indeed, not requiring the same dynamics would enable agents to imitate humans and robots with different morphologies, hence widely enlarging the applicability of II and alleviating the need for in-domain expert demonstrations.
This relaxed setting where the expert demonstrations comes from another domain has emerged as budding area with more realistic assumptions (Gupta et al., 2017: Liu et al., 2019; Sermanet et al., 2018; Kim et al., 2020; Raychaudhuri et al., 2021) that we will refer to as Crosy-Domain Imitation Learning. A common strategy of these works is to learn a mapping between the expert and agent domains. To do so, they require access to proxy tasks where both the expert and the agent act optimally in there respective domains. Under some structural assumptions, the learned map enables to transform trajectory in the expert domain into the agent domain while preserving the optimality. Although these methods indeed relax the typical setting of II., requiring proxy tasks heavily restrict the applicability of Cross-Domain IL. For example, it rules imitating an expert never seen before as well as transferring to new robot.
In this paper, we relax the assumptions of Cross-Domain IL and propose benchmark and method that do not need access to proxy tasks. To do so, we depart from the point of view taken by previous work and formalize Cross-Domait IL an optimal transport problem. We propose method, that
"a aarnaud. fick i nnger @berkeley.....ee.. edu, arnaudfica www.ouuficiiiggrr@eee f fo.coom Project site with videos and code: http://ammauddiciinngggghhhhuuu

---

Published as a conference paper at ICLR 2022
dE
PTE
PTA
Expert SE X AE SA
X AA
Agent
Figure 1: The Gromov- Wasserstein distance enables us to compare the stationary state-action distri- butions of two agents with different dynamics and state-action spaces. We use it as pseudo-reward for cross- domain imitation learning.
6
C 2
Figure 2: Isomorphic policies (definition 2) have the same pairwise distances within the state-action space of the stationary distributions. In Euclidean spaces, isometric transformations preserve these pairwise distances and include rotations, translations, and reflections.
call Gromov Wasserstein Imitation Learning (GWIL.), that uses the Maammov-Wasssesteinn distance to solve the benchmark. We formally characterize the scenario where GWIL preserves optimality (theorem 1), revealing the possibilities and limitations. The construction our proxy rewards to optimize optimal transport quantities using RL generalizes previous work that assumes uniform occupancy measures (Dadashi al., 2020; Papagiannis & Li, 2020) and is of independent interest. Our experiments show that GWIL learns optimal behaviors with a single demonstration from another domain without any proxy tasks non-trivial continuous control settings.
2 RELATED WORK
Imitation learning. An early approach to is Behavioral Cloning (Pomerleau, 1988; 1991) which amounts to training a classifier regressor via supervised learning to replicate the expert's demon- stration. Another key approach Inverse Reinforcement Learning (Ng Russell, 2000; Abbeel & Ng. 2004; Abbeel al., 2010), which aims learning reward function under which the observed demonstration is optimal and can then used to train a agent via RL. To bypass the need to learn the expert's reward function, Ho & Ermon (2016) show that IRI. is dual of an occupancy measure matching problem and propose an adversarial objective whose optimization approximately recover the expert's state-action occupancy measure, and a practical algorithm that uses a generative ad- versarial network (Goodfellow et al., 2014). While number of recent work aims at improving this algorithm relative the training instability caused by the minimax optimization, Primal Wasserstein Imitation Learning (PWIL) (Dadashi et al., 2020) and Sinkhorn Imitation Learning (SIL) (Papagian- nis & Li. 2020) view IL an optimal transport problem between occupancy measures to completely eliminate the minimax objective and outperforms adversarial methods in terms of sample efficiency. Heess al. (2017); Peng al. (2018); Zhu al. (2018); 8); Aytar et al. (2018) scale imitation learn- ing to complex human-lii like locomotion and game behavior in non-trivial settings. Our work is an extension of Dadashi et al. (2020); Papagiannis & Li (2020) from the Wasserstein the Gromov- Wasserstein setting. This takes us beyond limitation that the expert and imitator are in the same domain and into the cross-domain setting between agents that live different spaces.
Transfer learning across domains and morphologies. Work transferring knowledge between dif- ferent domains in RL typically learns a mapping between the state and action spaces. Ammar et al. (2015) unsupervised manifold alignment to find linear map between states that have similar
2

---

Published as a conference paper at ICLR 2022
local geometry but assume access to hand-crafted features. More recent work in transfer learning across viewpoint and embodiment mismatch leam state mapping without handcrafted features but assume access to paired and timealigned demonstration from both domains (Gupta et al., 2017; Liu et al., 2018; Sermanet et al., 2018). Furthermore, Kim et al. (2020); Raychaudhuri et al. (2021) propose methods to learn state mapping from unpaired and unaligned tasks. All these methods require proxy tasks, i.e. set of pairs of expert demonstrations from both domains, which limit the applicability of these methods real-world settings. Stadie al. (2017) have proposed combine adversarial learning and domain confusion to learn policy in the agent's domain without proxy tasks but their method only works in the case of small viewpoint mismatch. Zakka et al. (2021) take goal-driven perspective that seeks to imitate task progress rather than match fine-grained structural details to transfer between physical robots. In contrast, our method does not rely on learning an explicit cross-domain latent space between the agents, nor does it rely on proxy tasks. The Gromov- Wasserstein distance enables us to directly compare the different spaces without a shared space. The existing benchmark tasks are aware of assume access set of demonstrations from both agents whereas the experiments in our paper only assume access to expert demonstrations. Finally, other domain adaptation and transfer learning settings use Gromov-Wasse Wasserstein variants, e.g. for transfer between word embedding spaces (Alvarez-Melis & Jaakkola, 2018) and image spaces (Vayer et al., 2020b).
PRELIMINARIES
Metric Markov Decision Process. An infinite-horizon discounted Markov decision Process (MDP) tuple (S, A, R. P. ?o: where S and A are state and action spaces, P S A A(S) is the transition function, R: S x A + IR is the reward function, E ((S) is the initial state distribution and y is the discount factor. We equip MDPs with distance S A R and call the tuple (S, A, R. Po, 7, d) metric MDP.
Gromov-We Wasserstein distance. Let (X, (x,dx, "5,xxxxx (xx and (V. my) be two metric measure spaces, where dx, dy are distances, and H.,' fl are measures on their respective spaces". Optimal transport (Villani, 2009; Peyré al., 2019) studies how to compare measures. We will use the Gromov- Wasserstein distance (Mémoli, 2011) between metric measure spaces, which has been theoretically generalized and further studied in Sturm (2012); Peyré al. (2016); Vayer (2020) and is defined by
GW((X, Px). (V, dy.
min uEl((xx.))
|dx(1, - dy(v. s/)) ur. (1) x2
Σ
where U(ux, pby is the set of couplings between the atoms of the measures defined by
U(ux,Py)
RXX Vo E X, Σ Uny [x(x), Vy E EV, Σ Uz.v my(()) yEY
ZEX
gW compares the structure of two metric measure spaces by comparing the pairwise distances within each space to find the best isometry between the spaces. Figure 1 illustrates this distance in the case of the metric measure spaces (f x Ap, de, Pmw) and (SA x AA, d Paa).
4 CROSS-DOMAIN IMITATION LEARNING VIA OPTIMAL TRANSPORT
4.1 COMPARING POLICIES FROM ARBITRARILY DIFFERENT MDPs
For a stationary policy ग acting on a metric MDP (S, A, R, P,y,d), the occupancy measure is:
P= SSSAAA4 R
p(s,a) = 1000==(app)) (als)
P =
We compare policies from arbitrarily different MDPs in terms of their occupancy measures.
We use discrete spaces for readability but show empirical results in continuous spaces.
3

---

Published as a conference paper at ICLR 2022
Definition 1 (Gromov- Wassersttnnnnn distance between (isomorphic classes of) policies'), Given expert policy Tg and an agent policy A acting, respectively, on
ME (Se, AE, RE, PE, TE, dr) and MA (SA, AA. RA, PA, TA, dA)-
We define the Gromov-Wassersiee Wasserstein distance between and т as the Gromov- Wasserstein distance between the metric measure spaces (Se x AB,dE, Pag) and (S AA, dA, Pra)"
GW( GW((SR x AR,dE, Prg): (SA X AA, AA,dA,PaA)!
(2)
We now define isomorphisms between policies by comparing the state-action marginals and show that GW defines a distance between them. Figure 2 illustrates simple isomorphic policies.
Definition (Isomorphic policies). Two policies T and A are isomorphic if there exists a bijection supp[p=x] - supplpra] that satisfies for all (sg, ag); (sE', ,ag') E suppl Pngl and (SA. aA) supp|pra1:
dr ((SE, ar), (SF', a dA aF)) o(sE', Pa (A) = Pr (o ag))
(3)
(4)
In other words, & is isometry between (supp(ena) dg) and (supple=als d,A)) and Pea is the fush-forward measure dy (P(x))
Proposition 1. GW defines a metric the collection of all isomorphic classes of policies.
Proof. By definition 1, (WTTg,TA = 0 if and only if GW((Sg, dg. Pa ). (Sa, da. Pr )) 0. By Mémoli (2011, Theorem 5.1), this is if and only if there an isometry Ø supplessl supp|eas] such that PT^ (p(g))) By definition 2, this is true if and only if and TE are isomorphic. The symmetry and triangle inequality follow from Mémoli (2011, Theorem 5.1). L
The next theorem' gives sufficient condition recover, by minimizing GW, an optimal policy in the agent's domain up to an isometry.
Theorem 1. Consider two MDPs
ME (Sg. Ag. Rg, PE.PE.T) and MA - - AA, RA. PA-PA.7).
Suppose that there exists four distances ds. d d d defined on SE. Ag, SA and AE respectively, and two isometries (s aF, sgg) E SF. x Ar Sv the three following conditions hold:
(Sv, db.:)) + (5)) and v (AR, (AR,d5) + (As, d)) such that for all
R(sg,ag) RA(o(sE)) (((E))
(5)
Pe (8')) = P (0(8'))
pr(se) pa()(s)))
Consider optimal policy TH in Mg. Suppose that 7ow minimizes GW(T TGW) with dø (sg,ag) + (sE) + d (ax) and da (s), + d((ss) das)))) Then Tow is isomorphic to an optimal policy in MA
Proof. Consider the occupancy measure P'A SA AA + given by
(SA. + Px; (o 166-1(s)) (sA), ((x)))
We first show that PA is feasible in MA, i.e. there exists policy n' acting in MA with occupancy measure PA (a). Then we show that π is optimal in MA (b) and is isomorphic to wi (c). Finally show that NGW is isomorphic to A which concludes the proof (d).
P We later show that is actually
distance on policics but isomorphic classes of policies.
We always consider a policy in the context of the underlying metric MDP, such that every policy acting on dg * d.. This
(S, A, R, P. T, dg) are different from every policy acting (S, A, R, P,T, da) Wasserstein distance respects the identity of indiscernibles.
guarantees that the Gromov-Wasserteen
"Our proof is in finite statc-action spaces for readability and can be dircctly extended to infinite spaces.
A policy is optimal in the MDP (S, A, R, P. if it maximizes the expected return F E R(s,, az).
4

---

Published as a conference paper at ICLR 2022
(a) Consider 8 SA. By definition of PA
Σ (a(sA) Σ
Pag (sa( (aa))
E agEAg
Pr (o (sa),
Since Pug is feasible in M, it follows from Puterman (2014, Theorem 6.9.1) that
Σ Par (5 aF) pe(o (sA)) +7
+7 on RRERR,
Σ
PEsp,ax (6 (8A)) Pxh (85
GREAR
By conditions 6 and 7 and by definition PA
PE pE( +y
Σ
PE + PTg (SE
ABESE
= -PA(sA) +y
Σ
(SA) (A (1((s))
agESg.agEAg
= =PA(SA) +y
Σ
()))))(()) + (
follows that
Σ 6ΛEAA
+7
Σ
PAd a + pa)
*EEAA,AAAA
Therefore, by Puterman (2014, Theorem 6.9.1), Paa is feasible in MA i.e. there exists policy π acting in MA with occupancy measure p'x-
(b) By condition 7 and definition of P/x the expected return of x in MA then
Σ
AA
Σ
2 v (o.
Σ
PE ag ag) Rg(sE.0E)
AgES
Consider any policy NA in M'. By condition 7, the expected return of π is
Σ
PF (8 a as) = Σ
PF (0($)) ((a)) (E).
#A AA
RRESR,
Using the same arguments that we used show that Pa is feasible in M', we can show that (sg, + Prr (0(sg), (ax))
is feasible in M. It follows by optimality of T in M that
Σ
Pe (fae)
Σ
#EESE.RREAm
Σ
(((
AAAAARAAAA
follows that o is optimal in M'.
(c) Notice that
E: (SE, .F) + (b(sg), (((E))
isometry between (5 A dæ) and (SA AA, dA), where de and d and given, resp., by
(sE, ds + (# and (s) + (s dd(a)).
Furthermore, by definition, P2 {. Therefore by definition 2, - is isomorphic to n
5

---

Published as a conference paper at ICLR 2022
Algorithm Gromov-Wasserster imitation learning from single expert demonstration.
Inputs:
expert demonstration т, metrics on the expert (dg) and agent (d space Initialize the imitation agent's policy wo and value estimates V/
while Unconverged do Collect an episode T' Compute GW(T,T)
Set pseudo-rewards with eq. (9)
Update Te and Vi to optimize the pseudo-rewards
end while
(d) Recall from the statement of the theorem that Tow is minimizer of GW( Tow). Since T is isomorphic to g. it follows from prop. that GW( (Taa) 0. Therefore GW(T Tow) must be O. By prop. 1, it follows that there exists an isometry
X (suppípk)l dg) - (supp]pment 1.
such that Prew xa(Pg). Notice that X f lsupplo7] is an isometry from (supp(pA), to (supplpaaw d'a) and Prow (x o (mmppppppe)) isomorphic to A optimal policy in MA, which concludes the proof.
It follows by definition that now is
Remark 1. Theorem shows the possibilities and limitations of our method. It shows that our method can recover optimal policies even though arbitrary isometries are applied the state and action spaces of the expert's 's domain. Importantly, we don't 't need to know the isometries, hence our method is applicable to a wide range of settings. We will show empirically that our method produces strong results in other settings where the environment are not isometric and don't even have the same dimension. However, a limitation of our method that recovers optimal policy only up to isometries. We will that in practice running method different seeds enables to find an optimal policy in the agent's ss domain.
4.2 GROMOV- WASSERSTEIN IMITATION LEARNING
Minimizing GW between an expert and agent requires derivatives through the transition dynamics, which we typically don't have access to. We introduce a reward proxy suitable for training an agent's policy that minimizes GW via RL. Figure illustrates the method. For readability, combine expert state and action variables (sg, ag) into single variables 2g, and similarly for agent state-action pairs. Also, we define Z Sg x Ag and Z = SA AA.
Definition 3. Given an expert policy and an agent policy the Gromov- Wasserstein reward of the agent defined as rgw supplp==A + given by
1
rew(=a)
Σ
|dæ(ZE, *b)) - da(ZA, Z'A) "2g
(r(2A)
znEZR
where u" is the coupling minimizing objective
Proposition 2. If A minimizes GW(TE, AA), then A is an optimal policy for the reward rgw as defined in definition 3.
Proof. Suppose that TA minimizes GW(TE, TA), then by definition A maximizes
Σ RREZ
IdE(EE, #b)) (()2) "2
ZEEZE
PAEZA EZZ
(ZA)
Σ AA Csuppiexal
Σ
[dx(zx, 22) (a)
Par
(ZA)
2ge2g #gEZg 2/,E2A
Therefore, by Puterman (2014, Theorem 6.9.4), RA is optimal policy for reward rew.
n

---

Published as a conference paper at ICLR 2022
In practice approximate the occupancy measures of π by (s,
St a a.) 1(s E
EI=
where 7 = ($1, A1, ST, QT) is a finite trajectory collected with π. Assuming that all state-action pairs in the trajectory are different', á is uniform distribution. Given expert trajectory TE and an agent trajectory тΛ the (squared) Gromov-Wasserstein distance between the empirical occupancy measures is
&
(TE,TE) TA)= о
min
Σ
dg((s, - ()((
(8)
where O the set of is the of couplings between the atoms of the uniform measures defined by
@xxT {
0 E RTX
Vi [T],
Σ = 1/T,Vj [T),
Σ Ois = 1/T
JE[TT
EE|T]
In this case the reward is given for every state-action pairs in the trajectory by:
s8)) = -TA
Σ IT
[dE((s} a.)) (sF, da((s}) (s)
IS''Saa
where 6 is the coupling minimizing objective 8.
In practice we drop the factor T' because it is the same for every state-action pairs in the trajectory. Remark The construction of our reward proxy is defined for any occupancy measure and extends to previous work optimizing optimal transport quantities via RL that assumes uniform occupancy measure in the form of a trajectory to bypass the need derivatives through the transition dynamics (Dadashi et al., 2020; Papagiannis & Li, 2020).
Computing the pseudo-rewards. We compute the Saanov-Wasssesteinn distance using Peyré et al. (2016, Proposition and its gradient using Peyré al. (2016, Proposition 2). To compute the coupling minimizing 8, we use the conditional gradient method as in Ferradans et al. (2013).
Optimizing the pseudo-rewards. The pseudo-rewards obtain from GW for the imitation agent enable us to turn the imitation learning problem into reinforcement learning problem (Sutton & Barto, 2018) find the optimal policy for the Markov decision process induced by the pseudo- rewards. We consider agents with continuous state-action spaces and thus do policy optimization with the soft actor-critic algorithm (Haarnoja et al., 2018). Algorithm sums up GWIL in the case where a single expert trajectory is given to approximate the expert occupancy measure.
5 EXPERIMENTS
We propose benchmark set for cross-domain IL methods consisting of 3 tasks and aiming at an- swering the following questions:
1. Does GWIL recover optimal behaviors when the agent domain a rigid transformation of the expert domain? Yes, we demonstrate this with the maze in sect. 5.1.
2. Can GWIL recover optimal behaviors when the agent has different state and action spaces than the expert? Yes, we show in sect. 5.2 for slightly different state-action spaces between the cartpole and pendulum, and in sect. 5.3 for significantly different spaces between a walker and chectah.
To answer these three questions, (Todorov et al., 2012) and the DeepMind control suite (Tassa et al., 2018). We include videos of
simulated continuous control tasks implemented in Mujoco
7 We can add the time step to the state to distinguish between two identical state-action pairs in the trajectory. NNte that the Crroov--Wassenee distance defined in equ. (6) does not depend on the temporal ordering of the trajectories.
7

---

Published as a conference paper at ICLR 2022
(a) Expert
(h) Agent
Figure 3: Given single expert trajectory in the expert's domain (a), GWIL recovers optimal pol- icy in the agent's domain (b) without any external reward, as predicted by theorem 1. The green dot represents the initial state position and the episode ends when the agent reaches the goal represented by the red square.
Figure 4: Given single expert trajectory in the pendulum's domain (above), GWIL recovers the optimal behavior in the agent's domain (cartpole, below) without any external reward.
learned policies on our project site?. In all settings we use the Euclidean metric within the expert and agent spaces for dg and d.A-
5.1 AGENT DOMAIN RIGID TRANSFORMATION THE EXPERT DOMAIN
We evaluate the capacity of II. methods to transfer to rigid transformation of the expert domain by using the PointMass Maze environment from Hejna al. (2020). The agent's 's domain is obtained by applying reflection to the expert's maze. This task satisfies the condition of theorem 1 with ø being the reflection through the central horizontal plan and being the reflection through the z-axis in the action space. Therefore by theorem 1, the agent's optimal policy should be isomorphic to the policy trained using GWIL. By looking the geometry of the maze, is clear that every policy in the isometry class of an optimal policy is optimal. Therefore we expect GWIL to recover an optimal policy the agent's domain. Figure shows that GWIL indeed recovers an optimal policy.
5.2 AGENT AND THE EXPERT HAVE SLIGHTLY DIFFERENT STATE AND ACTION SPACES
We evaluate here the capacity of IL methods to transfer transformation that does not have to be rigid but description map should still be apparent by looking at the domains. A good example of such transformation is the one between the pendulum and cartpole. The pendulum is our expert's domain while cartpole constitutes our agent's 's domain. The expert is trained on the swingup task. Even though the transformation is not rigid, GWIL is able to recover the optimal behavior in the agent's domain shown in fig. 4. Notice that pendulum and cartpole do not have the same state-action
8

---

Published as a conference paper at ICLR 2022
Figure 5: Given single expert trajectory in the cheetah's domain (above), GWIL recovers the two elements the optimal policy's isometry class the agent's domain (walker), moving forward which is optimal (middle) and moving backward which is suboptimal (below). Interestingly, the resulting walker behaves like a cheetah.
space dimension: The pendulum has dimensions while the cartpole has 5 dimensions. Therefore GWIL can indeed be applied to transfer between problems with different dimension.
5.3 AGENT AND THE EXPERT HAVE SIGNIFICANTLY DIFFERENT STATE AND ACTION SPACES
We evaluate here the capacity of methods transfer non-trivial transformation between do- mains. A good example of such transformation is two arbitrarily different morphologies from the DeepMind Control Suite such as the cheetah and walker. The cheetah constitutes our expert's do- main while the walker constitutes our agent's 's domain. The expert is trained on the run task.
Although mapping between these two domains is not trivial minimizing the Gromov- Wasserstein solely enables the walker to interestingly learn move backward and forward by im- itating cheetah. Since the isometry class the optimal policy - moving forward- of the cheetah and walker contains suboptimal element -moving backward-, of these two trajectories. Indeed, depending on the seed used, GWIL produces a cheetah-imitating walker moving forward cheetah-imitating walker moving backward, as shown in fig. 5.
expect GWIL to recover one
CONCLUSION
Our work demonstrates that optimal transport distances are a useful foundational tool for cross- domain imitation across incomparable spaces. Future directions include exploring:
1. Scaling to more complex environments and agents towards the goal of transferring the structure many high-dimensional demonstrations of complex tasks into an agent.
2. The use gW to help agents explore extremely sparse-reward environments when we have expert demonstrations available from other agents.
3. How gW compares other optimal transport distances that work apply between two metric MDPs, such as Alvarez-Melis et al. (2019), that have more flexibility over how the spaces are connected and what invariances the coupling has.
4. Metrics aware the MDP's temporal structure such as Zhou & Torre (2009): Vayer et al. (2020a); Cohen et al. (2021) that build dynamic time warping (Müller, 2007). The Gromov- Wasserstein ignores the temporal information and ordering present within the trajectories.

---

Published as a conference paper at ICLR 2022
REFERENCES
Pieter Abbeel and Andrew Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp. 2004.
Pieter Abbeel, Adam Coates, and Andrew Y Ng. Autonomous helicopter aerobatics through ap- prenticeship learning. The International Journal of Robolics Research,
2010.
David Alvarez-Melis and Tommi S. Jaakkola. Croserv-waseesseiin alignment of word embedding spaces. In Empirical Methods in Natural Language Processing, 2018.
David Alvarez-Melis, Stefanie Jegelka, and Tommi S. Jaakkola. Towards optimal transport with global invariances. In Kamalika Chaudhuri and Masashi Sugiyama (eds.), Proceedings of the Secent International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pp. 1870-1879. PMLR, 16-18 Apr 2019. URL mlr press wawaalaaaeeeeeeeeeeaaeaaaaaa is s19a. 9t..hm..........................
https: httpps///prcceedingge
Haitham Bou Ammar, Eric Eaton, Paul Ruvolo, and Matthew E Taylor. Unsupervised cross-domain transfer in policy gradient reinforcement learning via manifold alignment. In Twenty-Nntth AAA/ Conference on Artificial Intelligence, 2015.
Yusuf Aytar, Tobias Pfaff, David Budden, Tom Le Paine, Ziyu Wang, and Nando de Freitas. Playing hard exploration games by watching youtube. arXiv preprint arXiv: /592, 2018.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemystaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, al. Dota with large scale deep reinforcement learning. arXiv preprint arXiv: (992106680) 2019.
Samuel Cohen, Giulia Luise, Alexander Terenin, Brandon Amos, and Marc Deisenroth. Aligning time series on incomparable spaces. In International Conference on Artificial Intelligence and Statistics, pp. 1036-1044. PMLR, 2021.
Robert Dadashi, Léonard Hussenot, Matthieu Geist, and Olivier Pietquin. Primal wasserstein imita- tion learning. arXiv preprint wrXiv:2006.04677 2020.
Sira Ferradans, Nicolas Papadakis, Julien Rabin, Gabriel Peyré, and Jean-François Aujol. Regu- larized discrete optimal transport. In International Conference Scale Space and Variational Methods in Computer Vision, pp. 428-439. Springer, 2013.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Abhishek Gupta, Coline Devin, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Learning invariant feature spaces to transfer skills with reinforcement learning. arXiv preprint arXiv:1703.029: 2017.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv: 182.v918/2.05905, 2018.
Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, SM Eslami, et al. Emergence of locomotion behaviours in rich environ- ments. arXiv preprint 3XXi:1707.0028880 2017.
Donald Hejna, Lerrel Pinto, and Pieter Abbeel. Hierarchically decoupled imitation for morphologi- cal transfer. In International Conference on Machine Learning, pp. 4159-4171 PMLR, 2020.
Jonathan Ho and S. Ermon. Generative adversarial imitation learning. In NIPS, 2016.
Kuno Kim, Yihong Gu, Jiaming Song, Shengjia Zhao, and Stefano Ermon. Domain adaptive imita- tion learning. In International Conference on Machine Learning, pp. 5286-5295. PMLR, 2020.
Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal Robotics Research, 1274, 2013.
10

---

Published as a conference paper at ICLR 2022
Sergey Levine. Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. The Journal of Machine Learning Research, 17(1):133-11373, 2016.
Fangchen Liu, Zhan Ling, Tongzhou Mu, and Hao Su. State alignment- baase imitation learning. arXiv preprint arXiv:1911.10947, 2019.
YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation: Learn- ing to imitate behaviors from raw video via context translation. In 2018 JEEE International Con- ference on Robotics and Automation (ICRA), pp. 1118-1122 IEEE, 2018.
Facundo Mémoli. Grsmov--asserstein distances and the metric approach to object matching. Foun- dations of computational mathematics, 11(4):417-487,200 2011.
Meinard Müller. Dynamic time warping. Information retrieval for music and motion, pp. 69-84, 2007.
Andrew Y. Ng and Stuart Russell. Algorithms for inverse reinforcement learning. In Proceedings of the Seventeenth International Conference on Machine Learning. ICML '00, pp. 663-670, San Francisco, CA, USA, 2000. Morgan Kaufmann Publishers Inc. ISBN 1558607072.
Georgios Papagiannis and Yunpeng Li. Imitation learning with sinkhorn distances. arXiv preprint arXiv:2008.09 2020.
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: Example- guided deep reinforcement learning of physics-based character skills. ACM Transactions on Graphics (TOG), 37(4):1-14, 2018.
Gabriel Peyré, Marco Cuturi, and Justin Solomon. Gromov-wasserstiin averaging of kernel and distance matrices. International Conference on Machine Learning, pp. 2016.
PMLR,
Gabriel Peyré, Marco Cuturi, al. Computational optimal transport: With applications data science. Foundations and Trends@ in Machine Learning, 11(5-6):355-60) 2019
D. Pomerleau. Alvinn: An autonomous land vehicle in a neural network. In NIPS, 1988.
D. Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural Computation, 3:88-97, 1991.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John
Wiley & Sons, 2014.
Dripta Raychaudhuri, Sujoy Paul, Jeroen van Baar, and Amit K Roy-Chowdhury. Cross-domain imitation from observations. arXiv preprint arXiv:2105. 10033 2021.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 1134-1141. IEEE, 2018.
Bradly C Stadie, Pieter Abbeel, and Ilya Sutskever. Third-person imitation learning. arXiv preprint 170..01703.01703, 2017.
Karl-Theodor Sturm. The space of spaces: curvature bounds and gradient flows on the space of metric measure spaces. arXiv preprini arXiv: 1208. 0434, 2012.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmalcki, Josh Merel, Andrew Lefrancq. et al. Deepmind control suite. arXiv preprint arXiv:1801. 00690, 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 8026-5033 IEEE, 2012.
11

---

Published as a conference paper at ICLR 2022
Titouan Vayer. A contribution to optimal transport incomparable spaces. arXiv preprint arXiv:2011.04447 2020.
Titouan Vayer, L. Chapel, N. Courty, Rémi Flamary, Yann Soullard, and R. Tavenard. Time series alignment with global invariances. ArXiv, abs/2002.03848,, 2020a.
Titouan Vayer, levgen Redko, Rémi Flamary, and Nicolas Courty. Co-optimal transport. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 77559-17570. Curran Asso- cc cc/paper/2020//ile Paper. ppf.
ciates, Inc., 2020b. URL
Cédric Villani. Optimal transport: old and new, volume 338. Springer, 2009.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czameekk, Michaël Mathicu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev. et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350-344 2019.
Kevin Zakka, Andy Zeng, Pete Florence, Jonathan Tompson, Jeannette Bohg. and De- bidatta Dwibedi. Xirl: Cross-embodiment inverse reinforcement learning. arXiv preprint arXiv:2106.0399 2021.
Feng Zhou and Fernando Torre. Canonical time warping for alignment of human behavior. Advances in neural information processing systems, 22:2286-2294, 2009.
Yuke Zhu, Ziyu Wang, Josh Merel, Andrei Rusu, Tom Erez, Serkan Cabi, Saran Tunyasuvunakool. János Kramár, Raia Hadsell, Nando de Freitas, al. Reinforcement and imitation learning for diverse visuomotor skills. arXiv preprint arXiv:1802.0955 2018
12

---

Published as a conference paper at ICLR 2022
A OPTIMIZATION OF THE PROXY REWARD
In this section we show that the proxy reward introduced in equation constitutes a learning signal that is easy to optimize using standards RL algorithms. Figure 6 shows proxy reward curves across 5 different seeds for the environments. We observe that in each environment the SAC learner converges quickly and consistently to the asymptotic episodic return. Thus there is reason think that the proxy reward introduced in equation will be similarly easy to optimize in other cross- domain imitation settings.
Gromov-Wasserstein Episodic Return
Isometric Mazes
Cheetah-Walker
Pendulum-Cartpale
0
-0.02
-0.04
-0.06
-0.08
-0.1
-0.12
-0.14
Samples
500k
1M
1.5M
Figure 6: The proxy reward introduced in equation 9 gives learning signal that is easily optimized using a standard RL algorithm.
B TRANSFER TO SPARSE-REWARD ENVIRONMENTS
In this section we show that GWIL can be used to facilitate learning in sparse-reward environments when the leamer has only access to expert demonstration from another domain. We compare GWIL to baseline leamer having access to a single demonstration from the same domain and minimizing the Wasserstein distance, as done in Dadashi al. (2020). In these experiments, both agents are given sparse reward signal in addition to their respective optimal transport proxy reward. We perform experiments
sparse-reward environment. In the first environment, the agent controls point mass in maze and obtain non-zero reward only if it reaches the end of the maze. In the second environment, which obtains a non-zero reward only if he can maintain the cartpole up for 10 consecutive time steps. Note that a SAC agent fails to learn any meaningful behavior in both environments. Figure 7 shows that GWIL is competitive with the baseline learner in the sparse maze environment even though GWIL has only access to a demonstration from another domain, while the baseline learner has access to demonstration from the same domain. Thus there is reason to think that GWIL efficiently and reliably extracts useful information from the expert domain and hence should work well other cross-domain imitation settings.
sparse version of cartpole, the agent controls a cartpole and
13

---

Published as a conference paper at ICLR 2022
Sparse Reward
Identical Mazes (Wasserstein)
Isometric Mazes (Gromov-Wasserstenn)
800
600
400
200
Samples
0
500k
1M
1.5M
Sparse Reward
Pendulum-Cartpole (ioomov-Wasserstein)
Cartpole-Cartplle (Wasserstein)
600
400
200
Samples
0
100k
200k
300k
400k
500k
Figure 7: In sparse-reward environments, GWIL obtains similar performance than a baseline learner minimizing the Wasserstein distance an expert the same domain.
C SCALABILITY OF GWIL
In this section we show that our implementation of GWIL offers good performance terms of wall- clock time. Note that the bottleneck of our method is in the computation of the optimal coupling which only depends on the number of time steps in the trajectories, and not on the dimension of the expert and the agent. Hence our method naturally scales with the dimension of the problems. Fur- thermore, while we have not used any entropy regularizer in our experiments, entropy regularized scale to demanding machine learn-
methods have been introduced enable Gromov-Wasserster ing tasks and can be easily incorporated into our code to further improve the scalability. Figure compares the time taken by GWIL. in the maze with the time taken by the baseline learner introduced in the previous section. It shows that imitating with Gromov- Wasseosttnse requires the same order of time than imitating with Wasserstein. Figure 9 compares the wall-clock time taken by a walker imitating cheetah using GWIL to reach walking speed (i.e., horizontal velocity of I) and the wall-clock time taken by a SAC walker trained to run. shows that a GWIL walker imitating a cheetah reaches walking speed faster than SAC agent trained to run. Even though the SAC agent optimizing for standing in addition to running, it was not obvious that GWIL could compete with SAC in terms of wall-clock time. These results gives hope that GWTL has the potential to scale to
14

---

Published as a conference paper at ICLR 2022
more complex problems (possibly with additional entropy regularizer) and be useful way to learn by analogy.
Sparse Reward
Identical Mazes (Wasserstein)
Isometric Mazes (Gromov-Wassersteen)
800
600
400
200
Time
0
10k
20k
30k
Figure 8: In the sparse maze environment, GWIL requieres the same order of wall-clock time than a baseline learner minimizing the Wasserstein distance to an expert in the same domain.
Horizontal Velocity
Walker Run (SAC)
Cheetah-Walker (Gromov-Wassersteii))
1.5
0.5
Time
500
1k
1.5k
2k
2.5k
Figure 9: A GWIL walker imitating a cheetah reaches a walking speed faster than a SAC walker trained to run in terms of wall-clock time.
15


Reviews:
[{"review_id": "kXVR0DG0agn", "reviewer": "Reviewer_tiAF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1.\tThis paper introduces and addresses an important and general cross domain imitation learning problem. \n2.\tAppling the Gromov-Wasserstein distance to align and compare two MDP domains provides insights to study cross domain imitation learning.\n3.\tThe proposed GWIL is novel. The authors also well justified limitations theoretically.\n\nWeakness:\n\n1.\tThere are only 3 tasks shown in the experiment section. More experiment results are preferred to show the effectiveness of the proposed solution. \n2.\tExperiment results are not well-visualized. It would be better to give a link showing the results in animation.\n", "summary_of_the_paper": "In this paper, the authors focus on a more general cross domain imitation learning problem where only expert demonstrations from one domain is available. To solve such a problem, the authors use the Gromov-Wasserstein distance to align and compare states between tasks from different domains and propose a Gromov-Wasserstein Imitation Learning (GWIL).  They also show theoretically the possibilities and limitations of GWIL. ", "summary_of_the_review": "In general, the paper addresses a general cross domain imitation learning problem where only expert demonstrations are available and proposes a novel GWIL. Though the experiment results are not visualized well, the work is highly likely to show new insights to researchers in imitation learning and domain adaptation domains. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Y7MXL6zWmx9", "reviewer": "Reviewer_pWm6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Good mathematical grounding\n- Further exploration of an interesting alternative to map optimal policies to new embodiments (possible practical applications)\n- Well written\n\nWeaknesses:\n- The main issue with this paper is the experimental evaluation. The presented results are just images of three cases. The images of the first case (Fig. 3) are hard to see. There is no numerical performance (success, reward, some degree of progress). This makes the experimental evaluation insufficient to understand the applicability of the presented approach. Add more experiments, numerical performance, different metrics…\n", "summary_of_the_paper": "This paper presents a method to transfer policies between different MDPs based on the minimization of Gromov-Wasserstein distance. This distance provides a pseudo-reward that can be used to learn via RL the optimal policy in the target MDP given an optimal policy in the original MDP. The method is optimal if the MDPs can be mapped into each other through an isometry, but works also empirically in other cases.\n", "summary_of_the_review": "In summary, while the paper presents an interesting turn on previously presented ideas, and the mathematical foundation is well worked out, the experimental evaluation is insufficient to support conclusions about the method.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Xwi87_F9Dz0", "reviewer": "Reviewer_KcRn", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThis looks to be a strong contribution, attempting to solve an important problem. The paper is well-written and relatively easy to follow, and results and proofs are interesting.\n\nQuestions:\n\nIt's unclear how easy an objective the proxy reward is to maximise. I would appreciate more clarity around this (eg. can you show some reward curves across multiple seeds/ runs), and convergence. The Gromov Wasserstein distance is quite an expensive distance to compute, can you comment on the computational feasibility of optimising eq 7 and potential scaling issues? \n\nRepeated mention of seed dependencies and effects is also concerning, I would appreciate more commentary on this.\n\nWhile I agree there are certainly settings where the Gromov Wasserstein distance makes sense from an imitation learning perspective, recovery up to an isometry can be prohibitive. eg. A human showing a drone how to take-off could result in a policy that lands drones, which is the opposite of what was demonstrated. I would value some discussion on these limitations - would it make more sense to optimise a different distance metric, or to use a different eg. non-euclidean kernel in settings like these? I suspect this is a non-trivial choice, that needs a substantial level of domain specific knowledge  - does this then run counter the original objectives of this work?\n\nMinor:\n\nPg 1. Intro is well written, but it would be great if Fig 2 could be shown earlier to give some more intuition into the Gromov Wasserstein distance and the solution framing.\n\nPg. 2 typo \"This takes us beyond limitation...\"\n\nIn existing imitation learning literature, much is made around limitations around learning from non-expert demonstrations. I would be interested to hear how the proposed approach would cope with these?\n\nEq 1 is in dire need of a figure to explain this.\n\nAs I understand it, although all proofs are provided in finite action and state spaces the proposed approach is said to scale to continuous spaces as ultimately it is only reliant on a suitable kernel function that can be expressed for continuous spaces. Is this correct?\n\n\"We will see that in practice, running our method on different seeds enables to find an optimal policy in the agent’s domain\". How do we know which seed produced the \"right\" policy?\n\nFig 3 needs improving - it is extremely hard to see the agent.\n\nFig 4/5. I'd love to see videos of these policies - did it actually learn to balance cartpole, or just to swing up? \n\n\n\n\n", "summary_of_the_paper": "This paper frames cross-domain imitation learning as an optimal transport problem using the Gromov-Wasserstein distance. This problem is highly relevant to imitation learning settings where there is often substantial domain mismatch between action and state spaces, eg. a humanoid robot learning to walk from a human demonstrator. The paper introduces a reward function that can be optimised and proves that this is equivalent to minimising the Gromov-Wasserstein distance between state action occupancies of an agent and expert. Substantial discussion/ proofs are included to show that minimising the Gromov-Wasserstein distance is equivalent to recovering an optimal policy up to an isometry. This is both a blessing and a curse, as it allows for optimal policies to be recovered under extreme changes in domain or differences, but does mean that recovered policies could be entirely unsuitable due to isometry. \n\nThe paper is well written and concisely written, although does get excessively mathy at times, when a figure could be more helpful. Experimental results corroborate the proofs and propositions, and highlight the value of the proposed approach.\n\n\n", "summary_of_the_review": "I enjoyed reading this paper, and think it adds greatly to the conversation around cross-domain imitation learning. The proposed approach has a number of strengths and limitations which I would appreciate hearing more about, particularly when it comes to convergence speeds, repeatability and computational requirements, but also whether strengths/weaknesses of optimal recovery of a policy up to an isometry have just shifted the need for specification of a mapping between expert and agent into a different domain.\n\n==== Post rebuttal comments ====\nThank you for engaging in the process, I still believe that this is a good paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MBtcP97wYYV", "reviewer": "Reviewer_ZsKi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- novel idea, to the best of my knowledge, for a difficult problem;  will inspire future work on \"learning by analogy\"\n- combination of theory, and empirical experiments;  I'm surprised by the extent the method works in practice\n\nWeaknesses:\n- with no learning curves presented, it is unclear if the cross-domain imitation learning actually provides a benefit,\n  for non-trivial systems, in terms of learning time or performance, as compared to learning from scratch.  It would be\n  beneficial to see these learning curves, and a wall-clock compute time comparison.\n- the limitations could be better articulated. The scalability is unclear (although it is unreasonable to expect the first \n   iteration of an idea like this to scale right away). The isometry constraint is likely to be limiting in many settings, \n  as is the choice of the Euclidean distance metric in the (state,action) space.\n- lack of an intuitive presentation of the Gromov-Wasserstein distance. I had to go elsewhere to obtain the intuition.\n  The actual method used to compute the GW distance, on discretely sampled trajectories.\n\nThe notation used for the GW sets-of-state action-pairs is confusing, i.e., \\tau,\nbecause the GW-distance is invariant with respect to temporal ordering (to my understanding), whereas the notation\nGW(tau, tau')seems to imply that the ordering needs to be preserved.  Perhaps introduce a different notation for\nthe data when the temporal information no longer needs to be preserved?\nThe connections of the trajectories / (s,a) data / occupancy measures needs to be better articulated for this reader.\n\nIs there a simple figure that could depict the essence of eqns 6 and 7?\n\nFigure 1 appears to simply show rotations.  If the goal is to show isometry for translations,\nit would be better to scatter the spirals more irregularly throughout the domain of the figure.\nSimilarly, why not include a reflection, as stated in the caption?\n\nIn Figure 3, the agents position is largely invisible.  Most of the readers may simply think there\nis an editing mistake and that the same figure was included 8 times.\n\nIn Figure 4, do the top and bottom row come from GW-corresponding state/actions?\n\nre: adding time to (s,a) to preserve uniqueness\nWouldn't this cause problems, given that the GW-distance would now include time?\n\n", "summary_of_the_paper": "A method is proposed for cross-domain imitation learning, without resorting to any form of correspondence.\nThis is done using a Gromov-Wasserstein distance between policies (in practice, Euclidiean distances on collected state-action pairs, \nwithin a given domain), which finds isometric transformations that best preserve distance measures, between the two domains.\nGiven an imitation domain and an expert domain with example trajectories, a pseudo-reward is computed \nbased on the degree to which the distances from a state to its neighbors in the imitation domain,\nare preserved in the expert domain. Given these pseudo-rewards, as computed for collected episodes,\nSAC is used as an RL algorithm to optimize the policy. \nThe paper contributes both a theoretical analysis and experiments with:  U-maze, pendulum-to-cart-and-pole, and half-cheetah-to-fallen-walker.\n", "summary_of_the_review": "The paper introduces a novel idea for imitation learning. \nIt likely has many limitations, but the idea of find suitable imitation-based correspondences is one that is being\npursued on multiple fronts, and this is a new approach, with a mix of theory and some initial proof-of-concept examples.\nThe paper could do better at explaining core ideas, and still needs learning curves in order to understand the benefit of the\ncross-domain transfer.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers suggested acceptance of the paper based on that the paper addresses an important problem and presents and validates interesting ideas for approaching it. Therea are some concerns regarding limited experiments - I'd like to encourage the authors to make an effort to address these concerns and also a few others raised in the reviews in the final version of their paper. The authors already made several updates to their paper in that regard during the discussion phase so I believe that the paper would be an interesting conttribution to the conference and I am recommending acceptance of the paper.
Decision: Accept (Poster)