Learn what matters: cross-domain imitation learning with task-relevant embeddings
Authors: Tim Franzmeyer
Philip Torr
Joao F. Henriques
Venue: NeurIPS 2022 Accept (2022)
Keywords: Reinforcement Learning
Learning from Observations
Inverse Reinforcement Learning
Imitation Learning
Domain Transfer

Abstract: We study how an autonomous agent learns to perform a task from demonstrations in a different domain, such as a different environment or different agent. Such cross-domain imitation learning is required to, for example, train an artificial agent from demonstrations of a human expert. We propose a scalable framework that enables cross-domain imitation learning without access to additional demonstrations or further domain knowledge. We jointly train the learner agent's policy and learn a mapping between the learner and expert domains with adversarial training. We effect this by using a mutual information criterion to find an embedding of the expert's state space that contains task-relevant information and is invariant to domain specifics. This step significantly simplifies estimating the mapping between the learner and expert domains and hence facilitates end-to-end learning. We demonstrate successful transfer of policies between considerably different domains, without extra supervision such as additional demonstrations, and in situations where other methods fail.

Learn what matters: cross-domain imitation learning with task-relevant embeddings
Tim Franzmeyer University of Oxford frtimOrobots, ox. ac. uk
Philip H. Torr University of Oxford philip torrdeng. ox. ac. uk
João F. Henriques University of Oxford
j joao@robots. ox. ac uk
Abstract
We study how an autonomous agent learns to perform a task from demonstrations in different domain, such different environment or different agent. Such cross- domain imitation learning is required to, for example, train an artificial agent from demonstrations of human expert. We propose scalable framework that enables cross-domain imitation learning without access to additional demonstrations or further domain knowledge. We jointly train the learner agent's policy and learn a mapping between the learner and expert domains with adversarial training. We effect this by using mutual information criterion to find an embedding of the expert's state space that contains task-relevant information and is invariant to domain specifics. This step significantly simplifies estimating the mapping between the learner and expert domains and hence facilitates end-to-end learning. We demonstrate successful transfer of policies between considerably different domains, without extra supervision such as additional demonstrations, and in situations where other methods fail.
Introduction
Reinforcement learning (RL) has shown great success in diverse tasks and distinct domains [3] however its performance hinges on defining precise reward functions. While rewards are straight- forward define in simple scenarios such as games and simulations. real-world scenarios are significantly more nuanced, especially when they involve interacting with humans.
One possibility for overcoming the problem of reward misspecification is to learn policies from observations expert behaviour, also known as imitation learning. Recent imitation learning algorithms rely on updating the learner agent's policy until the state occupancy of the learner matches that of the expert demonstrator [.] requiring the learner and expert to be in the same domain. Such requirement rarely holds true in
realistic scenarios. Consider for example the where robot arm learns to move an apple onto a plate from demonstrations of a human performing this task. Here, both domains do inherently share structure (the apples and the plates have similar appearances) but are distinct (the morphologies, dynamics and appearances of the two arms are different).
Enabling learner agent to successfully perform task from demonstrations that generated by different expert agent, which we refer to as a different domain even the tasks are related, would widely broaden the possibilities train artificial agents. This cross-domain imitation learning problem is seen as an important step towards value alignment, as it facilitates transferring behaviour from humans artificial agents (2 Chapter 7].
This problem has only been considered by researchers realistic settings recently. Due to its difficulty, previous work cross-domain imitation learning either assumes the expert's and learner's domains to be almost identical [42 [7 requires demonstrations of experts multiple domains
that are similar to the leamner' 45 relies the availability of demonstrations of proxy
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

MI
D
f
SE, 8'E
2, z!
D
SL, 12
2
L L
g
L
We consider a robot learning to place an apple onto a plate from demonstrations of human doing so. This illustrative cross-domain imitation learning problem requires finding the learner's s policy nı in its domain with states 8,. from demonstrations generated by the human expert (Dx) in the distinct expert domain with states 8g. We first use a mutual information criterion (fMI) to find an embedding function f that maps the expert state g to a task-relevant representation z to discard domain specific information. In the given example, f would primarily encode information about the apple and the plate, as these are most relevant the task. We next apply an adversarial loss D to jointly train all blue-shaded components, i.e., the policy of the leamner (TL),, the discriminator D and the mapping function which maps the learner states to the task-relevant representation of the expert domain. Here, the learner encoder maps the apple's color and the type of plate to that of the expert domain.
Figure 1:
tasks in both domains knowledge about both domains, since they have to be inherently similar to the target task to convey relevant mapping between domains [ Fickinger et al. [I0] overcome the need for proxy tasks by directly comparing distributions in both domains, effectively addressing the same problem setting as us. While very promising, applicability is limited to short demonstrations and Euclidean spaces.
B00 |
Designing such proxy tasks is manual process that requires prior
We propose to jointly learn the learner policy and the mapping between the learner and expert state spaces, utilizing adversarial training. Unlike standard generative adversarial imitation learn- we use domain-specific encoders for both the learner and expert. We therefore devise mutual information criterion find expert encoder that preserves task-relevant information while discarding domain specifics irrelevant to the task. Note that in general, cross-domain imitation learning an under-defined problem, as a unique optimal policy for the learner is not defined as part of the problem: for example, should a humanoid agent that imitates a cheetah crawl (imitating its gait) walk (moving the same direction)?
ing I6 39)
We evaluate our cross-domain imitation learning approach in different cross-embodiment imitation learning scenarios, comparing relevant benchmarks, and find that method robustly learns policies that clearly outperform the baselines. We conduct several ablation studies, particular finding that we can control how much domain-specific information is transferred from the expert- effectively interpolating between mimicking the expert's behaviour as much as possible and finding novel policies that use different strategies to maximize the expert's reward.
Our contributions are:
We propose a mutual information criterion to find an embedding of the expert state which contains task-relevant information, while discarding domain specifics irrelevant to the task.
We learn the mapping between the leamner domain and the tele-aelev embedding without additional proxy task demonstrations.
We demonstrate training robust policies across diverse environments, and the ability to modulate how information flows between the learner and expert domains.

---

2 Related Work
Imitation learning considers the problem of finding optimal policy for a learner agent from demonstrations generated by an expert agent, where inverse reinforcement learning (IRL) [I 46 recovers a reward function under which the observed expert's behaviour is optimal. More recent works [6.
39 define imitation learning as a distribution matching problem and use adversarial
training 14 to directly find the learner's policy, without explicitly recovering the expert's reward.
dross-domain imitation learning generalizes imitation learning to the case where the learner and expert are in different domains. Small mismaches between the domains, such changes in viewpoint or gravitational force, or small variations of the dynamics, are addressed by 22 Bohez et al. 6 To learn policies cross-domain in the presence of larger mismatches, such different embodiments of the learner and the expert, previous works used demonstrations of proxy tasks to learn mapping between the learner and expert domain, which is then used to find the learner's optimal policy [5 23
36 g and
BO [& utilized latent embedding of the environment state 25 14 or assumed the reward signal be given [4] GWIL [TO] does not rely proxy tasks and minimizes the distance between the state-action probability distributions of both agents which lie in different spaces
35
2 This approach assumes Euclidean spaces and is computationally intractable when
using longer demonstrations, which generally improve the performance of learning algorithms when available.
Our approach obviates the need for proxy tasks, scales to detailed demonstrations of complex behaviours, and enables the control of how much domain- information is transferred to the learner domain.
In classical RL 126 where behaviour is learned from given reward function, mutual information objectives are commonly used to find compact state representations that increase performance by discarding irrelevant information 29 24 22 We propose to similarly learn representation
of the expert's 's state that contains task-relevant information while being invariant domain specifics.
3 Background
Definitions. Following Kim et al. [18), define domain as a tuple (S, {). where S denotes the state space, A is the action space. P is the transition function, and Ç is the initial distribution over states. Given an action
A. the distribution over the next state is given by the transition function as (Ps''s,, a). An infinite horizon Markov decision process (MDP) is defined by adding a reward function S x A R, which describes a specific task, and discount factor > [0, to the domain tuple. We define the expert agent's MDP as ME = (SE,AE PE, TE, YE, SE), and its policy as a map T Sg + B(Aa), where B is the of all probability measures on Ag. We define the learner MDP Mn learner policy π analogously, except that the learner MDP has no reward function or discount factor. An expert trajectory is sequence of states TE (5 st s .ggh where n denotes the length of the trajectory. We denote Dg {Ti} to be a set of such trajectories.
Problem Definition. The objective of cross- domain imitation learning is to find policy ay that optimally performs task in the learner domain MM, given demonstrations Dg in the expert domain Mg. In contrast to most prior work, we do not assume access to a dataset of proxy tasks taske-simple primitive skills in both domains that are similar but different from the inference task-to be given. We do not assume access to the expert demonstration's 's actions, which may be non-trivial to obtain, e.g.. when learning from videos or human demonstrations, and therefore consider the expert demonstrations to consist only of states.
A Adversarial Imitation Learning from Observations. We first consider the equal- case in which both MDPs are equivalent, i.e., -M.MMPr. ME, and assume that the expert agent's 's optimal policy Fæ under Tæ is known. Torabi al. [9 define a solution to this problem as an extension of the standard imitation learning problem [66] by minimizing the divergence between the learner's
State-transition distribution Paz and that of the expert Pag. as
arg min - (TL.) + Dass (Prz (8, (8,88) 8¹) - Prg 8')) ))- RL IRI. (Tæ),
(1)
L
3

---

where Dus is the Jensen- Shannon divergence and ( ) is the learner's policy entropy Statettranstion distribution for policy defined as
46 The
Pa ( 83)
Σ a) Σ P(s: P(se 8
In particular, the expert's state-transition distribution Prx: is estimated using expert demonstrations Dr. The above objective (eq. D can also be derived as the composition of the IRL and RL problems, where TE (TF) denotes the solution the Inverse Reinforcement Learning problem from policy TE and n = RL(rE) denotes the solution to the RL problem with reward rg.
The IRL component, which recovers the reward function the expert's demonstrations are uniquely optima by finding a reward function that assigns high rewards the expert policy and low rewards to other policies, is given as IRL(TE) arg min, (max,, E [r(s, s')l [rg,r(sss')]
S S - I under which
4 Unsupervised Imitation Learning Across Domains
We first introduce the cross-domain imitation learning problem before deriving an adversarial learning objective that allows the simultaneous training of the learner's policy and mapping between the MDPs of the learner and expert. We then demonstrate how the cross-domain imitation learning problem can be significantly simplified be finding embedding of the expert agent's state space that contains task-relevant information while discarding domain-specific aspects. Lastly, we introduce a inve-invariance constraint prevent degenerate mapping solutions. As our approach does not rely on additional demonstrations from proxy tasks, we refer to as unsupervised cross-domain imitation learning objective (UDIL).
4.1 Cross-domain ad adversarial imitation learning
We consider the case in which the expert's and agent's MDPs are different, i.e.. ML Mg. such as when learner and expert are of different embodiments or are in different environments. Kim et al. 18 show that, if there exists an injective mapping that reduces the learner MDP Mr to the expert MDP Mg, g, then a policy æ that optimal in Mr is also optimal in the Me-
Since we do not assume extra supervision from the expert's actions, we define the mapping function between the learner and expert MDPs g Sr. SE as a mapping between the respective state spaces. We accordingly define the cross-domain adversarial imitation objective as
arg min - (TL) + Das (Pra. (g(#t), g((()))()))))))))))))) - Pag (sg, z))
Z2
Applying the mapping g to the learner agent's 's state allows us to compare the learner's and expert's distributions, even though they are defined over different state-spaces.
4.2 Reducing the expert's 's state dimension
The full state of the expert domain 8g generally contains information that is specific to the task which the expert is demonstrating, defined by the expert's 's reward function TE, as well as information that is specific to the domain but irrelevant to the task itself. We simplify the cross-domain imitation learning problem by reducing the expert agent's 's state space to a task-relevant embedding that is invariant to domain specifics.
We assume that the learner state is multi-dimensionne and recall the IRL component of the adversarial imitation problem (eq. []. which finds the reward function under which the expert's 's behavior is optimal. We define a second mapping function ʃ SF. - Z, that maps the expert states 8 E SE to lower-dimensional representations E Σ, with |2| < |S When f is chosen as a dimension reduction operation that discards state dimensions of which the reward is independent, we can write the IRL component of eq. I as function of only the embedded representation (proof in app. 8.1.1
I We swap cost function for the reward function and omit the cost function regularization for simplicity. We assume that the reward function is also defined the embedding space Z, see app. 8.1.1 for details.

---

as
[RL( = arg min max Ex [r(2, - Exp (r(,)
(4)
Simplifying the mapping between learner and expert. Assuming f to be given, can further
redefine the mapping between leamer and expert state g S2 + Z. That is, the state transformation g no longer has map the learner state the full expert state, but only to the task-relevant embedding of the expert state. This not only significantly simplifies the complexity of the mapping function but also prevents transferring irrelevant domain specifies from the expert to the learner domain. We can then rewrite the cross-domain adversarial imitation objective as
arg min H( + Dus (pnz. (g($L),,($())) Paee ((()))
(5)
which minimizes the distance between the transformed distribution over learner states s and the distribution over embedded expert states 2.
4.3 Finding a task-relevant embedding
We now detail how to find a embedding function f from the expert demonstrations Dg. We first assemble set containing all expert transitions ('g,,g) observed in the trajectories of the demonstration set Dg. We then generate a set of pscudo-random transitions ($rand, sannd) by independently sampling two states of all individual states contained DE- We then model all state transitions ss) and their corresponding labels y. indicating whether it is a random or expert transition, as realizations of random variable (S, , Y) on SE x SE x {0, 1}. Note that any time-invarian embedding f SE + Z induces a random variable (2, on 2 x 2 x {0, via then define the mapping f as a mapping that maximizes the mutual information / between the label Y and the embedded state transition (Z, Z'), that is,
(Z, 2') = ((S))))("))
arg max I ((Z2Z'); = arg max [(Lf(S)
(6)
f
Observe that maximizing (Z;Y) would lead to non-informative representations, as the states contained in the random trajectories are indeed states of the expert trajectory; only state transitions (S, can distinguish between the two.
4.4 Avoiding degenerate solutions
Jointly learning the mapping function and the learner agent's policy TL may lead to degenerate mappings if g is function of arbitrary complexity. An overly-expressive g can make the divergence between distributions arbitrarily small, regardless of their common structure, by the universality property of the uniform distribution, i.e., any two distributions can be transformed into each other by leveraging their cumulative density functions (CDFs) and inverse CDFs. We prevent these degenerate solutions with an information asymmetry constraint: we ensure that the mapping f is inverinvariant, while the JS-divergence compares distributions across time, i.e., in time-variant manner. A theoretical analysis presented in app. 8.1.2
4.5 Unsupervised cross-domain adversarial imitation learning
We finally define the unsupervised cross-domain adversarial imitation learning (UDIL) objective as an adversarial learning problem. We iterate between updating the learner agent's policy ni the mapping between the learner's s and expert's ' state spaces, and the discriminator D. The discriminator's objective is distinguish between state transitions generated by the learner and state transitions generated by the expert, giving the overall objective
min max Ex (Dog((D(()))) + Eng [lo[[1 - Do(2,
Experiments
Preliminaries. We test our approach on two different benchmarks that represent multiple do- mains and different agents with both environment-hasee and agent- based tasks. We designed our experiments to answer the following questions.
5

---

Gripper From Loegstick
Longstick from Gripper
20-
Stegs
Supp
1730. XIRI
agents with different embodiments (such Gripper and Longstick displayed here) have to move the three magenta-coll colored blocks to the magenta-shaded target zone at the top of the environment. We evaluate the reward achieved by both learner agents when trained on demonstrations of the other using either our algorithm UDIL, or the XIRI. [45 baseline.
Figure 2: In the XMagical benchmark 40 55
Can find task-relevant embeddings of the expert state solely from expert demonstrations, and improve the performance of imitation learning?
Does the proposed framework robustly learn meaningful policies compared previous work?
Can we control the amount of domain-specific information transferred from the expert to the learner?
We compare with the GWIL baseline 1O. which is the only other work that makes similar assump- tions to ours, i.e., unsupervised cross-domain imitation learning with access only to demonstrations of a single expert agent. In the later presented XMagical environment, we also compare to a modi- fied single-demonstrator-agge multiple distinct expert agents. As no reward function in the learner domain is given, we measure performance of the learner agent by defining its reward as the components of the expert agent's reward function that can be directly transferred to the learner domain. To ensure reproducibility. we all experiments random seeds zero to six, report mean and standard error for all experiments (lines and shaded areas), and describe the experiments in full detail in appendix section 8.2
version of XIRL
45 which originally relies demonstrations of
5.1 XIRL baseline
Setup. Figure 2 shows the XMagical environment 41 55
which consists of four agents with different embodiments that have perform equivalent modifications in the environment, namely pushing all blocks to a shaded region. The corresponding baseline algorithm XIRL (45] trains each agent with demonstrations of the three other expert agents. As our work only requires demonstrations from a single expert agent, we focus on the two most distinct agents, Gripper and Longstick, which are displayed Figure 2). and evaluate the performance of each when trained on demonstrations of the other. The reward is given as function of the average distance between the task-relevant objects and their target positions.
Finding task-relevant embedding. The environment state in XMagical is given multidi- mensional vector that describes different absolute and relative positions of environment objects and the agent itself. To find the task-relevant embedding of this state we first generate sets of expert and pscudo-random transitions, as described in section As maximizing mutual information objectives in large continuous domains intractable
we instead approximate the objective in eq. (6) by first computing the empirical mutual information between state transitions and labels for each individual state dimension, using the method of Ross BII. We then find the meskvreleaant embedding by selecting the dimensions with highest mutual information using the elbow method [19]. We find a clear margin between those state dimensions that are intuitively relevant to the task, such as dimensions that describe the positions of the blocks, and those dimensions that are intuitively domain-specific and less relevant the task, such as dimensions that describe the position of the robot.
Imitation learning with task-relevant embedding of the expert state. We use the dataset of expert demonstrations provided by Zakka et al. 45 to compare the performance of our approach to that of the XIRL baseline. We follow Zakka et al. 55 and likewise use the simplified imitation

---

Figure 3: Sample rollouts from the three agents hopper, halfcheetah and walker (section 5.2 We illustrate locomotion strategies learned for different dimensions of the expert state's 's embedding space (see discussion in section 5.3). Right side: For larger d, the hopper performs a swimming like movement (top). For smaller d (bottom), the hopper is straight and propels itself forward using only its foot. Left Side: For smaller d, the halfcheetah propels itself forward with its front the ground (top). For larger d, the walker performs mix of a falling and walking motion (bottom).
learning framework where the learner agent simply receives a reward signal that corresponds to the distance between the current environment state and the target environment state, which is pre- computed by averaging over all terminal states contained in the set of expert demonstrations. Note that the main difference between UDIL and XIRL is the rell-relevant embedding of the expert state: XIRL relies on the full expert state. We use the XIRL implementation as given by the authors, apply directly to the state space and do not change any parameters. Figure shows that we consistently outperform XIRI and in both cases achieve a score close to the maximum possible. We find that our method obtains task-relevant embeddings of the state from expert demonstrations alone, which significantly improves performance of cross-domain imitation learning in the XMagical environment.
5.2 Cross-domain imitation learning of robot control
We now evaluate UDIL in the complex Mujoco environments
We use the embodiments displayed in Figure S hopper, walker and halfcheetah, which are commonly used to evaluate (cross-domain) imitation learning algorithms 120 16
38
30 We use the fixed-length trajectory implementation [13] of these environments to prevent implicitly rewarding the learner agent for longer trajectories; the significance of this effect demonstrated in Kostrikov et al. 20 We first find a minimal task-relevant embedding, investigate the performance, and compare GWIL. We then conduct ablation studies to evaluate the importance of the individual components of our framework and investigate how the transfer of information from the expert to the learner domains be controlled by varying the size the task-relevant expert embedding. We provide videos of the resulting behaviour, as described in in appendix 8.4
I2]
Finding fask-relevant embedding. Analogously to the previous section 5.1 we first generate sets of expert and pscudo-randee transitions, and compute the mutual information between individual state dimensions and the transition labels. We find that across all three agents, the I position of the torso has highest task-relevance, followed by the position (height). This intuitively makes sense, the expert agents receive relatively large rewards during training for moving in the positive x direction, followed by a smaller reward for being in a healthy (upright) position [7]. Note here that these findings are derived only from the expert demonstrations, without any knowledge of the rewards. Hereafter, the dimensions which describe the angular positions of the main joints with respect to the torso have highest mutual information; lowest mutual information is found for state dimensions that describe velocities of sub-components. We identify the task-relevant embedding with the elbow method as the positions that deseribe the torso, and later conduct ablation studies with larger embeddings.
Jointly learning the learner's policy and mapping function. We parameterize the learner encoder such that it learns affine transformation of the input and define its loss as the negative of the
7

---

Hopper from HalfCheetah
HalfCheetah from Hopper
Walker from Hopper
10
Reward
D
01
50
0.0
2.5
0.0
2.5
0.0
2.5
Environment Steps
x10%
Environment Steps x10 HalfChectah from Walker
Environment Steps x10
Hopper from Walker
Walker from HalfCheetah
10
Reward
01
0-
0.0
2.5
0.0
2.5
0.0
2.5
Environment Steps x100
Environment Steps x x10
Environment Steps x x10
UDIL
GWIL
Figure 4: Reward curves for cross-domain imitation learning for different combinations of learner and expert agents. The mean performance is shown as a solid line, and the standard deviation as shaded area.
discriminator's loss, i.e., the learner encoder trained fool the discriminator. The policy of the learner is parameterized by a neural network, which, in contrast to the learner encoder, cannot be trained by backpropagating the discriminator loss as a sampling step required obtain the state transitions form the learner policy. We follow Ho and Ermon [16] and train the learner policy with RL, with the learner agent receiving higher rewards for taking actions that result in transformed state transitions g( z)). g(((,) which are more likely to fool the discriminator D, i.e., which are more likely to be from the expert's rask-relevant state-transition distribution Pe (2g, ='E)) We use DAC (0], to jointly train original implementation to ensure comparability. We define the reward of the learner agent as the distance covered in the target direction, as this is the only reward component that is common among all three agents, and compare performance to GWIL ICC
and D. as depicted in Figure | and do not alter any hyperparameters given in the
Results. Figure 4 shows that the learner agents robustly learn meaningful policies for six random initializations across different combinations of expert and learner. We find that the hopper and walker cover about 50% of the distance compared to when they trained with their ground truth rewards, with the halfcheetah achieving about 13% of the expert distance.
We qualitatively inspected the behaviours learned by the agents and found novel locomotion strategies that are distinct from those the expert. We illustrate these strategies in Figure 3 We hypothesize that these new behaviours investigate in section 5.3 how the embedding size can be chosen to transfer more information from the expert to the learner. It can be in Figure that our framework consistently outperforms the GWIL baseline; although we tried different hyperparameter configurations, we found the results of GWIL be highly stochastie, which is due the properties of the used, as indicated by the authors GWIL [TO] Remark 1].
enabled by the task-relevant embedding of the expert state and further
distance 25
5.3 Ablation Studies
We present our ablation studies that clarify the importance and influence of the different components of the framework, focusing on the hopper and halfcheetah agents.
Varying the dimension of the task-relevant embedding. We investigate the relevance of the task- relevant state embedding's 's dimension d and hypothesize that for larger embeddings, more information transferred from the expert the learner domain. We evaluate the performance as well as the resulting agent behaviours for
(3, 6, all), where all refers no reduction, i.c. f is an identity

---

Hopper from HalfCbectah
HalfCheetah Hopper
16.0
Rowan
2.5-
0-
0.0-
6.0
0.0
25
Favimnmen Steps
Flvinemmnnt Steps
UDIL
time invariant
embeedling
larger embhedding
Figure 5: Achieved reward (travelled distance) by both hopper and halfcheetah, when trained on demonstrations of the other with different ablations of our framework. See section 5.3 for details.
mapping, in which case the learner encoder g has to map the full learner state space the full expert state space. We can observe in Figure D that the mean performance and robustness generally decrease when increasing the embedding size. We investigate different locomotion strategies adopted by the learner agent, dependent on the embedding size d. and illustrate these in Figure 3 We found that for d = 3, both hopper and halfcheetah would lie down on the floor and propel themselves forward. For larger embeddings d E {6, all), both would adopt strategies more similar to the demonstrations by lifting their torso off the ground for longer. The hopper would hop for a few moments and then perform swimming-lin movement, the halfcheetah would exhibit animal-like quadruped gait.
We conclude that changing the size of the expert's state embedding allows us modulate the transfer of information between the expert and the leamnr domains. In one extreme, one might want the learner to solve a task with a minimal task-relevant embedding. strategies distinct from the expert, which could for example allow it to outperform the expert. In the other extreme, one might want the learner to replicate the strategies of the expert as closely as possible, which could be useful if the learner fails to solve the task with less information. Choosing the size the task-relevan embedding then trades off between these two options.
allow the learner to develop
Omitting the time invariance constraint. We omit the time-inarriance constraint by reducing the discriminator input from s, s' to just the current state s. While this setting yields successful results in same-domain imitation learning (27]. we found the time-invariance constraint to be essential for adversarial cross-domain imitation learning (see Figure
Learning from a single trajectory. We investigated the performance of our approach when only a single expert trajectory is given, which represents the most direct comparison to GWIL, GWIL can only utilize a single expert trajectory due to its computational complexity. We find that UDIL likewise outperforms GWIL by large margin if only one demonstration is given, and show more results in appendix 8.3.3
Conclusion
We introduce a novel framework for cross-domain imitation learning, which allows a learner agent jointly learn to imitate expert and learn mapping between both state spaces, when they dissimilar. This is made possible by defining mutual information criterion to find a task-relevant embedding of the expert's state, which further allows to control the transfer of information between the expert and learner domains. Our method shows robust performance across different random instantiations and domains, improving significantly upon previous work. However, as cross-domain imitation learning is generally an under-defined problem, the risk of learning incorrect policies remains. The mutual information objective used to find the task-relevant embedding might yield degenerate solutions in special cases, such as when the expert's policy induces uniform distribution over state transitions, or when the environment only partially observable. Also, finding the ideal size of the task-relevant embedding might be challenging in more complex domains. Similarly, the application of our algorithm to high-dimensional observation spaces requires further contributions and may constitute an interesting direction for future work.

---

7 Acknowledgements
We thank Dylan Campbell and Jakob Foerster for their helpful feedback. We are also grateful to the anonymous reviewers for their valuable suggestions. This work was supported by the Royal Academy of Engineering (RF) 18008199,81..... 163).
References
[1] Abbecl, P. and Ng. A. Y. [2004), Apprenticeship learning via inverse reinforcement learning, in 'Proceed- ings, Twenty-Firsl International Conference on Machine Learning, ICML 2004'.
[2] Akkaya, I,, Andrychowicz, M., Chociej, M., Litwin, M., McGrew, B., Petron, A., Paino, A., Plappert, M.. Powell. G., Ribas, R. and others [2019], 'Solving rubik's cube with robot hand', arXiv preprint arXiv:911000713.
[3] Anand, A., Racah, E., Ozair, S., Bengio, Y., Côté, A. and Devon Hjelm, [2019]. Unsupervised state representation learning in atari, in 'Advances in Neural Information Processing Systems', Vol. 32.
[4] Arora, S. and Doshi, P. [2021], 'A survey inverse reinforcement learning: Challenges, methods and progress', Artificial Intelligence 297, 103500.
[5] Belghazi, M. I., Baratin, A., Rajeswar, S., Ozair, S., Bengio, Y., Courville, A. and Hjelm, R. D. [2018], Mutual information neural estimation, in '35th International Conference Machine Learning, ICML 2018', Vol. 2.
[6] Bohez, S., Tunyasuvunakool, S., Brakel. P.. Sadeghi, F., Hasenclever, L., Tassa. Y., Parisotto, E.,, Humplik, J., Haarnoja, T., Hafner. and others [2022], 'Imitate and Repurpose: Learning Reusable Robot Movement Skills From Human and Animal Behaviors' arXiv preprint 2rXiv:2203.77338
[7] Brockman, G., Cheung. V., Pettersson, L., Schneider, J., Schulman, J., Tang. J. and Zaremba, W. [2016]. "Openai gym', arXiv preprint 1606.0040000
[8] Cetin, E. and Celiktutan, O. [2021], 'Domain-robust visual imitation learning with mutual information constraints', arXiv preprint arXiv:2103.0507.
[9] Cover, T. M. and Thomas, J. A. [2005], Elements of Information Theory.
[10] Fickinger, A., Cohen, S., Russell, S. and Amos, B. [2021]. 'Cross-Domain Imitation Learning via Optimal Transport', arXiv preprint 100.042.....684.
[11] Fu, J., Luo, K. and Levine, S. [2017], 'Leaming robust rewards with adversarial inverse reinforcement learning', arXiv preprint arXiv:1710.11248.
[12] Gangwani, T. and Peng, J. [2020]. 'Statc-only imitation with transition dynamics mismatch', arXiv preprint arXiv:2002.11879.
[13] Gleave, A., Freire, P., Wang, S. and Toyer, S. [2020], 'seals: Suite environments for algorithms that learn specifications', htttps://githu/ com wwt.//ooo//ooooeee http////annnmmmaiilllee///eeeeee
[14] Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D. Ozair. S., Courville, A. and Bengio. Y. [2014], Generative adversarial nets, in 'Advances in Neural Information Processing Systems". Vol. 3.
[15] Gupta, A., Devin, C., Liu, Y., Abbeel, P. and Levine, S. [2017], 'Leannigg invariant feature spaces transfer skills with reinforeement learning', arXiv preprint / 703.02299.
[16] Ho, and Ermon, S. [2016], 'Generative adversarial imitation learning', Advances neural information processing systems 29.
[17] Hudson, E.. Warnell, G., Torabi, F. and Stone, P. [2021], 'Skeletal feature compensation for imitation learning with embodiment mismatch", arXiv preprint arXiv:2104.07800.
[18] Kim, K., Gu, Y., Son, J., Zha. S. and Ermo, S. [2020], Domain Adaptive Imitation Learning, in '37th International Conference on Machine Learning, ICML 2020', Vol. PartF168147-7,
[19] Kodinariya. T. M. and Makwana. P. R. [2013], 'Review on determining number of cluster in k-means clustering", International Journal 1(6), 95-95.
10

---

[20] Kostrikov. I,, Agrawal, K. K.. Dwibedi, D., Levine, S. and Tompson, J. [20]8a], 'Discriminator- critic: Addressing sample inefficienccy and reward bias in adversarial imitation learning'. arXiv preprint urXiv:1809.022 0.09.02925.
[21] Kostrikov, I,, Agrawal, K. K., Dwihedi, D., Levine, S. and Tompson, J. [2018b], Discriminator-actor critic: Addressing sample inefficiency and reward bias in adversarial imitation learning'. arXiv preprint 1.09..80000222
[22] Lee, A. X., Nagabandi, A., Abbeel, P. and Levine, S. [2020], Stochastic latent actor-critic: Deep reinforce- ment learning with a latent variable model, in 'Advances in Neural Information Processing Systems', Vol. A02e-becember.
[23] Liu, F., Ling, Z., Mu, and Su, H. [2019]. 'State alignment- based imitation learning', arxiv preprint vrXiv:1911.10%4
[24] Mazoure, B., des Combes, R. T., Doan, I., Bachman, and Hjelm, R. D. [2020]. Deep reinforcement and InfoMax learning, in 'Advances Neural Information Processing Systems', Vol. 0020-ercmmber
[25] Mémoli, F. [2011]. dations of Computational Mathematics 11(4).
Wasserseenn Distances and the Metric Approach Object Matching' Foun-
[26] Montague, P. [1999], 'Reinforcement Learning: An Introduction, by Sutton, R.S. and Barto, A.G.', Trends in Cognitive Sciences 3(9).
[27] Orsini, M.. Raichuk. A.. Hussenot, L., Vincent, D., Dadashi, R., Girgin. S., Geist, M., Bachem, O., Pietquin. O. and Andrychowicz, M. [2021], 'What matters for adversarial imitation learning?', Advances in Neural Information Processing Systems 34.
[28] Radosavovic, I., Wang, X., Pinto, L. and Malik, J. [2020], State-only imitation learning for dexterous manipulation. in '2021 IEEE/RS] International Conference on Intelligent Robots and Systems (IROS). pp. 7865-7871.
[29] Rakelly, K., Gupta, A., Florensa, C. and Levine, S. [2021]. "Which Mutual-Informatee Information Representation Learning Objectives are Sufficient for Control?", Advances in Neural Information Processing Systems 34.
[30] Raychaudhuri, D. S., Paul, S., Vanbaar, J. and Shy--hhwyh A. K. [2021], Cross-domain imitation from observations, in 'International Conference Machine Learning', pp. 8902-8912.
[31] Ross, B. C. [2014], 'Mutual information between discrete and continuous data sets', PLoS ONE 9()..
[32] Russell, S. [2019], Human compatible: Artificial intelligence and the problem of control, Penguin.
[33] Satopaa, V., Albrecht, J., Irwin, D. and Raghavan, B. [2011], Finding a" kncedle" in haystack: Detecting knee points in system behavior, in '2011 31st international conference on distributed computing systems workshops', IEÉE, pp. 166-171.
[34] Schmeckpeper, K., Rybkin, O., Daniilidis, Levine, S. and Finn, C. [2020], 'Reinforcement learning with vidcos: Combining offiine observations with interaction arXiv preprint arXiv:201 1.06507.
[35] Sermanet, P., Lynch, C., Chehotar, Y., Hsu, J., Jang, E., Schaal, S., Levine, S. and Brain, G. [2018], Time-contrastive networks: Self-supervised learning from video, in 22018 IEEE international conference on robotics and automation (ICRA)', pp. 11 1134-1141.
[36] Stadie, C., Abbeel, P. and Sutskever, [2017], "Third-person imitation learning'. arXiv preprint 1YXiv:170..01700
[37] Stooke, A., Lee, K., Abbecl, P. and Laskin, M. [2021], Decoupling representation leamning from reinforce- ment learning, in 'International Conference on Machine Learning', pp. 9870-9877
[38] Todorov, E., Erez, T. and Tassa, Y. [2012], MuJoCo: A physics engine for model-based control, in 'IEEE International Conference on Intelligent Robots and Systems'.
[39] Torabi, F., Wamell, G. and Stone, P. [2018), "Generative adversarial imitation from observation', arXiv preprint 1807008007
[40] Toyer, S., Shah. R., Critch, A. and Russell. S. [2020a], 'The magical benchmark for robust imitation'. Advances in Neural Information Processing Systems 33, 18295.
[41] Toyer, S., Shah, R., Critch, A. and Russell, S. [20206], "The magical benchmark for robust imitation', Advances in Neural Information Processing Systems 33, 12284-1829)

---

[42] Viano, L.. Huang. Y.-T., Kamalaruban, P., Innes, C., Ramamoorthy, S. and Weller, A. [2022]. 'Robust learning from observation with model misspecification" arXiv preprini arX1v:2202.0600:
[43] Vinyals, O., Babuschkin, I, Czarnecki, W. M. Mathieu, M., Dudzik, A., Chung. J. Choi, D. H., Powell,
R., Ewalds, T., Georgiev, P., Oh, J., Horgan, D.. Kroiss, M., Danihelka, 1., Huang, A.. Sifre. L... Cai, T., Agapiou. P. Jaderberg, M., Vezhnevets. A. S., Leblond, R., Pohlen, T., Dalibard, V., Budden. D., Sulsky, Y., Molloy, J., Paine, T. L., Gulcehre, C., Wang, Z., Pfaff, T., Wu, Y., Ring, R., Yogatama, D., Wünsch, D.,
McKinney, K., Smith, O., Schaul, T., Lillicrap. T., Kavukcuoglu, K., Hassabis, D., Apps, C. and Silver, D. [2019], 'Grandmaster level in StarCraft II using multi-agent reinforcement learning', Nature 575(7782).
[44] Yin, Z.-H., Sun. L., Ma. H., Tomizuka, M. and Li, W.-J [2021]. "Cross Domain Robot Imitation with Invariant Representation', arXiv preprint 21t.:22999999490
[45] Zakka, K., Zeng, A., Florence, P., Tompson, J., Bohg. J. and Dwibedi, D. [2022], Xirl: Cross-embodiment inverse reinforcement learning, in 'Conference on Robot Learning,, pp. 537-546.
[46] Ziebart, B. D., Maas. A. L.. Bagnell, A. and Dey, A. K. [2008], Maximum entropy inverse reinforcement learning,, in 'Aaai', Vol. 8, Chicago, II,, USA, pp. 1433-1438.
12


Reviews:
[{"review_id": "VX4wzQPy9Sc", "reviewer": "Reviewer_YPHf", "summary": "The problem setting under consideration is cross-domain imitation learning, where the goal is to enable an imitation learning agent to learn from expert demonstrations from a different environment or agent embodiment. The authors propose a method for cross-domain imitation learning that requires less supervision than prior methods (i.e. without additional proxy tasks or multiple demonstrators/domains), by primarily leveraging a mutual information-based objective to encourage learning a more task-relevant representation of the expert state space. This embedding is then used by the learner for an imitation learning objective. The method further imposes a time-invariance constraint to prevent learning a degenerate embedding space, and the overall method uses an adversarial imitation learning setup to learn solely from observations. The experiments compare against other recent methods in cross-domain imitation learning, across a range of task settings. ", "questions": "- Have the authors considered / experimented with using multiple expert agents providing demonstrations? One of the stated motivations for this work is wanting to avoid reliance on a large number of different demonstrators, but it would be interesting to see if there is any performance improvement in the multi-demonstrator case (i.e. would the quality of the learned embedding space improve, or is one demonstrator sufficient for capturing the task-relevant features?). \n- In the ablations plot (Figure 5) with Hopper from HalfCheetah, what is the intuition behind the noisy reward curve for UDIL? As adversarial training objectives can be unstable to train, I am curious if the authors have seen any other similar instabilities / difficulties with training.\n- Do the authors have any hypotheses for why the performance increases then decreases for the larger embedding case for Hopper from HalfCheetah in Figure 5?", "limitations": "The paper would be improved with a section more clearly discussing the limitations of the proposed approach -- e.g. if there are difficulties with adversarial training, whether there are cross-domain environments or tasks where the proposed mutual information objective would fail, or how much additional overhead is required to search for the best embedding dimension size for the task you care about.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Observational cross-domain imitation learning has grown increasingly relevant as a problem setting, for which the authors propose a novel self-supervised method (to the best of my knowledge). While the individual components -- mutual information based representation learning, adversarial imitation learning -- are not novel, the construction of the method and problem domain seems novel. This problem setting and proposed method has high relevance to the research community, as methods developed in this area opens the door for unsupervised learning of behaviours from widely available online demonstrations (eg. Youtube videos).\n- The method is well explained and straightforward. The authors carry out a range of evaluations and ablations to show the capabilities of their proposed method, comparing against recent work in this domain, and show that their proposed method performs well. \n- The paper is very well written, structured nicely, and was a pleasure to read. \n- Being able to adjust how much of the task-relevant information to retain using the embedding size is an interesting outcome of the method. \n\nWeaknesses:\n- While not explicitly directed at imitation learning across different embodiments, there are some relevant works in unsupervised methods for domain regularization in observational imitation learning, which should be cited in the paper: \n  - Stadie, Bradly C., Pieter Abbeel, and Ilya Sutskever. \"Third-person imitation learning.\" (2017).\n  - Cetin, Edoardo, and Oya Celiktutan. \"Domain-robust visual imitation learning with mutual information constraints.\" (2021).\n- While the performance in Section 5.1 compared to XIRL seems strong, I would like to see the same evaluations carried out on different combinations of agent embodiments rather than just the two that are most different. In the other embodiment cases, does UDIL still outperform XIRL or is the performance more similar as the embodiment gap closes? \n- It also seems like the authors do not use the adversarial imitation learning setup in the comparisons against XIRL. It would be interesting to see if the adversarial setup improves or hurts performance in this case.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Iwa3uyYtQnm", "reviewer": "Reviewer_LDkQ", "summary": "The authors propose to address cross-embodiment imitation learning by using GAILfO but with a learned cross-embodiment embedding.  They learn the mappings from (i) expert state space to the embedding space, and (ii) learner state space to the embedding space by respectively (i) using a mutual-information objective with state transition pairs and psuedorandom state transition pairs, and (ii", "questions": "Please see my discussion of Weaknesses for questions.\n", "limitations": "The authors mention only that \"the risk of learning incorrect policies remains\".  I encourage the authors to consider and discuss other limitations that I raised in the Weaknesses discussion.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. The formulation is pretty clean, satisfying, and nice to follow.\n\n2. The results show that the authors are able to get the method to work pretty well on the challenging problem of cross-embodiment imitation.\n\n## Weaknesses\n\n1. Discussion of relation to existing works\n  - Issue: The discussion of prior work is kind of annoyingly pedantic, and doesn't have to be, in attempting to carve out a novelty statement about what other methods can or can't do.  In general, paragraph 4 of the intro (continuing onto page 2), and the \"cross-domain imitation learning\" discussions in the Related Work, and the brief mention of related work in the Abstract, are all annoying. For example, they say that [40] needs multiple demonstrator agents, but as they show themselves, it is pretty trivial to just use [40] but with a single demonstrator agent.  Also, [9] is another prior work that effectively addresses the same problem statement.  Also, some of the limitations of it, such as the footnote on page 2, also apply to the presented work.  To be honest, I'm not sure the authors are able to say any type of clean statement about a type of scenario that their algorithm can address that others can't, and I think they know that.  But that's okay!  That doesn't mean their work isn't interesting or isn't useful.\n  - Suggested solution: I think the authors would do well to focus on: rather than trying to come up with statement of what other methods can't address (which has some issues as noted above), they might do well to just instead focus on how their proposed formulation is *different*, how it is interesting, and how at least in the shown experiments it may also work better.  Additionally, they could be more positive towards, and appreciative of, the prior works in this challenging subfield, which have helped carve a path that they can follow and attempt to improve upon.  Rather than their current final sentence on lines 81-84, they could basically just say \"We propose a different formulation using X, Y, and Z, and further show that compared to [40] and [9], empirical results suggest our method is more capable than these prior works on the tested settings.\"  Or something like that.\n\n2. Is \"cross-domain\" better terminology to be using here than \"cross-embodiment\"?  All of the experiments are focused on cross-embodiment imitation learning, rather than any other notion of difference between expert and learner.  I don't think it matters too much either way, but as the authors themselves say, the cross-embodiment case is probably in general the most challenging, and hence why they focus on it.\n\n3. Probably a different title should be necessary?  Per my discussion of point 1 above, there are actually prior works which address the cross-embodiment imitation learning setting, and similarly do so with the same amount of supervision as the present work (only expert demonstrations needed from a different embodiment).  Accordingly, I don't think the title is very truthful or accurate.  It seems to suggest that this paper introduces the idea of doing cross-domain imitation learning, which as discussed in point 1, isn't the case.  Further, it's not really unsupervised... there isn't really such a thing as unsupervised imitation learning.  Specifically, the expert is providing supervision.  A more accurate and useful title for the community might be something like \"Generative Adversarial Cross-Embodiment Imitation with Task-Relevant Embeddings\" or something like that.  That title actually calls out how the paper addresses a known problem, but with a different method.\n\n4. Probably should take one of the contributions off the list?  Specifically the first stated contribution, \"We devise a framework to learn the mapping between the learner and expert domains in an unsupervised fashion, i.e., without additional proxy task demonstrations.\" also applies to for example [40] and [9], so I don't think it can be claimed as a contribution of the present work.  However, I think the other 2 contributions in the list are strong and sufficient.\n\n5. Some cases where the proposed algorithm definitely fails. For example consider a case where of simple environment where an agent visits every state, in a particular order, exciting every possible state-next-state $(s, s')$ transition an equal number of times.  In this case, the task-relevant embedding would fail to do anything useful, since although the long-horizon order may be considered what matters, instead the algorithm only uses state-next-state pairs to find the task-relevant embedding.  Although this is a simple case, it may in general point to problems for using the proposed algorithm on long-horizon, multi-step tasks.\n\n6. Some cases where the proposed algorithm I think would fail.  On this point, I may be wrong, but I just don't see how specifically the learning of $g$ (the mapping from learner state to shared embedding space) is required to learn anything useful.  I just don't see what's stopping $g$ from deciding that it wants to completely scramble the ordering of all states visited.  While Section 4.4 discusses avoiding a degenerate mapping for $f$, it doesn't discuss it for $g$.\n\n7. Some missing key related work.  Another very related work is \"Reinforcement Learning with Videos\" (https://arxiv.org/pdf/2011.06507.pdf).  Although I think it still significantly different from the proposed work, for example not adopting the GAIL style formulation, it does also address cross-embodiment imitation and also learns a shared embedding space.  Note too that the v2 November 2021 version actually corrected a prior mistake which was that, contrary to the published conference version, the method does not actually need paired data.  Accordingly it has, like the other works mentioned too, basically the same data requirements as the presented work.\n\n8. Issue with time-invariant mapping not discussed.  This related to my point #6.  If f is time-invariant, it can't know about histories of states mattering in any particular way, which makes long-horizon demonstrations with potentially repetitive tasks not addressable.  Also, they don't discuss how the actually make it time invariant.  It already looks time invariant in Section 4.3 -- seems like they don't need to do anything else to make it so.\n\n9. Note that the comparison between the present work and Kim et al [17] could be improved.  Basically the present work discards [17] because it needs some proxy tasks.  But could the authors describe why specifically they need proxy tasks?  This is an especially important comparison because Kim et al generally falls in the category of doing cross-embodiment generative advsersarial imitation.  So the details really matter here with the comparison, I think.\n\n\n## Minor\n\n1. Line 23 says that \"classic\" imitation learning algorithms do state occupancy matching between learner and expert... this though is the formulation of GAIL?  I'm not sure I would call GAIL a \"classic\" IL method yet?\n\n2. Eq. 5 would probably be even more clear if rather than $z_E$ it was called out that this is $f(s_E)$.  Then it's clear that both expert and learner state spaces are being mapped to some shared embedding space.\n\n3. Error: line 186, first symbol, should be $g$.\n\n4. Lines 104-106 -- they make it seem like an extra-special thing to not know the actions of the expert, but in the cross-embodiment case, the learner has different actions anyway, so it's kind of already implied that the actions won't be that useuf.\n\n5. Line 109 -- Torabi et al, GAILfO, is a good paper, but as those authors would probably admit themselves, as is highlighted in the name of their algorithm, it is a pretty minor modification to GAIL.  Accordingly, it seems justified to give a citation to GAIL as well as GAILfO on line 109.\n\n6. Error: line 154 should say $g: S_L \\rightarrow Z_E$.\n\n7. May consider just using $Z$ for notation, rather than $Z_E$, since the embedding space is used by both learner and expert, rather than just expert.\n\n8. Line 230-233 is confusing -- who is \"we\" here?  Are you describing how you modify [40], or what the proposed algorithm uses?\n\n9. Line 241 -- I'm not sure anybody would call hopper, walker, and half-cheetah \"high-dimensional\"?  I think walker has a 3D action space, and the others are 6D?  6D isn't small... it's moderately hard... but I don't think \"high-dimensional\".", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FTaTgTMQwfi", "reviewer": "Reviewer_KU89", "summary": "The paper presents UDIL, unsupervised cross-domain imitation learning, a method for learning a policy in one environment using expert demonstrations from another environment. The focus is on learning a task-relevant embedding, which identifies which parts of the state should be used for mapping between learner and expert data. Experiments are presented in two domains: the XMagical benchmark (in which the learner and expert are circular and a long stick or vice versa) and MuJoCo (in which the agents are hopper, halfcheetah, or walker). UDIL outperforms XIRL in the XMagical domain and outperforms GWIL in the MuJoCo domain.", "questions": "I wonder if there are additional baselines that could be used. For example, what happens if vanilla imitation learning is done on the task-relevant embedding? I expect this would work poorly since the policy does need some environment-specific information. Another baseline could be some form of oracle in which the task-relevant dimensions are hand-picked or make use of the reward in some way.\n\nMinor (no need to respond):\n* Typo on line 293 (missing period after domain).\n* Typo on line 318 (3 p's in appendix).", "limitations": "I'd like to see a discussion of limitations, perhaps addressing how the method might extend to more complex observations/environments.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* The paper is well-written and well-motivated. The method and design decisions are explained clearly.\n* The analysis of the size of the task-relevant embedding (dimension d) is interesting and original.\n\nWeaknesses\n* The experiments are limited to environments with small states, where there is a clear distinction between task-relevant dimensions and dimensions which can be discarded. The claims would be strengthened if experiments were extended to environments with visual observations (e.g. atari flavors) or at least more nuanced ones (perhaps MiniGrid).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept