Out-of-Dynamics Imitation Learning from Multimodal Demonstrations
Authors: Yiwen Qiu
Jialong Wu
Zhangjie Cao
Mingsheng Long
Venue: CoRL 2022 Poster (2022)
Keywords: Imitation Learning
Out-of-Dynamics Imitation Learning

Abstract: Existing imitation learning works mainly assume that the demonstrator who collects demonstrations shares the same dynamics as the imitator. However, the assumption limits the usage of imitation learning, especially when collecting demonstrations for the imitator is difficult. In this paper, we study out-of-dynamics imitation learning (OOD-IL), which relaxes the assumption to that the demonstrator and the imitator have the same state spaces but could have different action spaces and dynamics. OOD-IL enables imitation learning to utilize demonstrations from a wide range of demonstrators but introduces a new challenge: some demonstrations cannot be achieved by the imitator due to the different dynamics. Prior works try to filter out such demonstrations by feasibility measurements, but ignore the fact that the demonstrations exhibit a multimodal distribution since the different demonstrators may take different policies in different dynamics.
We develop a better transferability measurement to tackle this newly-emerged challenge. We firstly design a novel sequence-based contrastive clustering algorithm to cluster demonstrations from the same mode to avoid the mutual interference of demonstrations from different modes, and then learn the transferability of each demonstration with an adversarial-learning based algorithm in each cluster. Experiment results on several MuJoCo environments, a driving environment, and a simulated robot environment show that the proposed transferability measurement more accurately finds and down-weights non-transferable demonstrations and outperforms prior works on the final imitation learning performance. We show the videos of our experiment results on our website.

Out-of-Dynamics Imitation Learning from Multimodal Demonstrations
Yiwen Qiu', Jialong Wu?, Zhangjie Cao², Mingsheng Long2 2 Department of Automation, Tsinghua University. China Schooo of Software, BNRist, Tsinghua University, China
Department of Computer Science, Stanford University, Stanford, CA 94305, USA com
{qywmei, wuwjiallgoo2222 caoz stanford. edu, mingsheng@tsingguu.. edu. cn
Abstract: Existing imitation learning works mainly assume that the demonstrator who collects demonstrations shares the same dynamics as the imitator. However, the assumption limits the usage of imitation learning, especially when collecting demonstrations for the imitator is difficult. In this paper, we study out-of-dynamics imitation learning (OOD-IL), which relaxes the assumption to that the demonstrator and the imitator have the same state spaces but could have different action spaces and dynamics. OOD-IL enables imitation learning utilize demonstrations from wide range of demonstrators but introduces new challenge: some demonstrations cannot be achieved by the imitator due the different dynamics. Prior works try filter out such demonstrations by feasibility measurements, but ignore the fact that the demonstrations exhibit a multimodal distribution since the different demonstra- tors may take different policies in different dynamics, which hinders learning an accurate measurement. We develop a better transferability measurement to tackle this newly-emerged challenge. We first design novel sequence based contrastive clustering algorithm cluster demonstrations from the same mode avoid the mutual interference of demonstrations from different modes and then learn the transferability of each demonstration with an Sntversarill-eerniie based algorithm in each cluster. Experiment results several MuloCo environments, driving en- vironment and simulated robot environment show that the proposed transferability measurement more accurately finds and down- weights non-transferable demonstra- tions and outperforms prior works on the final imitation learning performance. We show the videos of our experiment results on our website.
Keywords: Imitation Learning, Out-of-Dynamics Imitation Learning
Introduction
Imitation learning is widely-used policy learning paradigm solve robotics and control tasks, which learns the policy from demonstrations [1, 2]. Standard imitation learning assumes that the demonstrator who collects the demonstrations shares the same dynamics with the imitator, which means that their state spaces, action spaces and the transition models are all the same [3, 4, 5]. However, such strict assumption limits the practical usage of imitation learning, especially when it is difficult to collect demonstrations in the imitator's ' environment.
In this paper, we relax the assumption to that the demonstrator and the imitator share the same state space but the dynamics could be different, i.e., their action spaces and the transition models could be different. We name the new imitation learning setting out-of-dynamic imitation learning (OOD-IL). The new assumption poses fewer requirements on the demonstrator and enables imitation learning to utilize broader range of demonstrations, For example, in autonomous driving, to learn the policy of an autonomous vehicle in a new city, we may use the database of driving behavior of human vehicles containing useful information instead of manually collecting demonstrations on the autonomous vehicle. But such a database cannot be used by standard imitation learning since the vehicles have different dynamics and the city environments be different. Furthermore, in real
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Initator's Enviroument
Sampling by Pw
Inforece
Demonstation
Transferable
Source Demonstrates
Database
Damonstrations
(a) Problem setting and the general framework.
Multimodal distribution.
Figure (a) We are given demonstrations collected from an unknown mixture of demonstrators, which exhibit multimodal distribution since different demonstrators may take different policies. Some demonstrations (red) may not be achievable by the target agent while others (green) are achiev- able. Learning from these unachievable demonstrations introduces unknown behaviors. We develop transferability measurement to mitigate negative impact of these non-transferable demonstrations and learn from transferable ones. (b) A I-SNE visualization of demonstrations collected from 4 dynamics in the MuJoCo environment.
applications, it is often difficult and expensive to obtain enough data from a single source, thus it is beneficial to utilize data from a mixture of massive data sources for better real-world performance.
As shown in Figure (()), OOD-IL introduces a challenge that the mixture of demonstrations collected in different dynamics may not be achievable by the imitator, and thus non ton-traasferable. The ofate-of-the- thethe-rrt work constructs feasibility measurement to down- down-weight the non-transion nrn-transferable demon- strations by learning unimodal policy from feasibility (F-MDP) [6]. However, shown in Figure (b), we notice that the prior work overlooks an important fact: the demonstrations collected from different demonstrators exhibit a multimodal distribution since different demonstrators may take different policies. The multimodal demonstrations make it difficult to learn a unimodal policy from the f-MDP as well learn an accurate feasibility measurement. Furthermore, f-MDP suffers from slow and inaccurate optimization difficulties due to a step-by-step optimization procedure.
In this paper, we address OOD-IL with the above challenges by developing better transferabilityy measurement to determine how transferable each demonstration is for the imitator. To fully remove the interference from multimodal distribution, we design sequence- algorithm to simultancously learn a hidden space and cluster the trajectories the hidden space, which ensures each cluster introduces unimodal trajectory distribution. Then learn the trans- ferability for each cluster respectively by generative adversarial imitation learning (GAIL) [5], a much easier-to-one tostti-to-opiimize objective than f-MDP. The discriminator in GAIL distinguishes transitions from the demonstrators' distribution to the imitators' distribution, which serves as the transferability measurement indicate the likelihood for demonstration trajectory be reproduced by the imita- tor. Experiment results on several MuJoCo environments, a driving environment, and a simulated Franka Panda Arm environment show that by reweighting demonstrations with the proposed transfer- ability measurement, the final imitation learning policy outperforms all baselines and achieves the thtt-off-the-ar performance.
contrastive clustering
Related Works
Standard Imitation Learning. Imitation learning learns policy to imitate the behavior in demon- strations. Existing algorithms can be roughly categorized into three types: Behavior Cloning (BC), Inverse Reinforcement Learning (IRL), and Generative Adversarial Imitation Learning (GAIL). BC utilizes supervised learning to directly learn the policy from date-action pairs [7]. Following works propose dataset aggregation [3] or policy aggregation [8, 9] address the compounding errors prob- lem in BC. Torabi et al. [10] try to learn from state sequences by first recovering the actions through an inverse dynamics model and then conducting behavior cloning. IRL first recovers the reward function from demonstrations and then learns the policy using the learned reward [11, 12, 4, 13]. GAle-bassed works match the occupancy measure between the learned policy and the demonstrations through adversarial learning to seek the optimal policy [5, 14]. Also, GAIL is demonstrated to suc-

---

cessfully imitate state sequences [15, 16, 17]. However, all the methods assume that the demonstrator and the imitator share the same dynamics, which violates the assumption of OOD-IL.
Out-of-dynamics Imitation Learning. To address the OOD-IL problem, there are works learning a correspondence model [18] between the demonstrator and the imitator [19, 20, 21, 22, 23). However, these methods assume that strict correspondence, i.c. a one-to-one mapping between the state spaces and action spaces, exists between the demonstrator and the imitator. This assumption can be violated in real-world applications, e.g., a 7-DoF robot arm and a 3-DoF robot arm have no such correspondence since some behavior of the 7-DoF robot cannot be realized by the 3-DoF robot. Recent works relax the assumption to that the demonstrator and the imitator only share the state space [24, 25]. One line of work aims maximally follow the demonstrations [24, 26, 27, 28], but following the non-transferable demonstrations is impossible.
Cao et al. [25] develop feasibility measurement dewn-weight non-transferable demonstrations by learning an inverse dynamics model, but the learned inverse dynamics may not generalize to all the demonstrations. To address this difficulty, the state-of-the-art work improves the feasibility by learning the optimal policy in f-MDP [6]. However, f-MDP suffers from two limitations. Firstly, f-MDP enforces one unimodal policy maximally imitate the demonstrations but the demonstrations with multimodal distribution are hard to be modeled by such a policy. Secondly, at each time step, only when the policy optimized in all prior time steps can it be optimized to maximize the reward at the current time step. Such inappropriate design makes it difficult and inefficient to learn the optimal policy of f-MDP. We instead use the discriminator in GAIL learn the transferability measurement.
Contrastive Clustering, Unsupervised deep clustering methods with the aid of contrastive represen- tation learning have been proposed for computer vision [29, 30, 31] and natural language processing [32]. To decompose the intrinsic multimodal distribution of demonstrations, we also adopt the intuition of jointly learning representations and clustering in an end-to-end manner. While prior works either focus on visual representations in the form of a single fixed-length vector [30, 31J or rely on data augmentations with heavy computational burden [32], our Sequence-based Contrastive Clustering is unique in handling trajectory data with various sequential lengths. by utilizing simple and efficient down-sampling to construct positive pairs of contrastive learning.
3 Out-of-dynamics Imitation Learning
In OOD-IL. we aim to learn policy for the agent of interest (the imitator) from demonstrations
collected from multiple demonstrators with different dynamics from the imitator. Formally, we model both the demonstrators and the imitator
a standard Markov decision process (MDP)
R, Po: 7). Here S is the state space, A is the action space, p: S x A S + [0. 1] M = (S, A. p, R, is the transition probability and R S S + R is the reward function, which is solely defined on states and shared between the demonstrators and the imitator. Such design aims to satisfy the basic requirement of imitation learning that the demonstrators and the imitator should finish the same task [24, 25). 7 is the shared discount factor. The action spaces A and the transition probability p of different demonstrators and the imitator are different. In particular, we use M' (S, A', ,p',, R. po- 7) to indicate the MDP of the imitator. A policy for the imitator n' SSx A + [o, 1] defines a probability distribution over the action space in a given state. The policy is evaluated by the expected return. which is defined by 7- = T 12re0 7 RR;, $t41)], where indicates the time step.
We formalize the policy learning
imitation learning problem where the reward function R is unknown. We aim to learn the policy x² from a set of demonstrations collected by different demonstrators = = (f1. } where each trajectory is a sequence of states & {sg sy s...... s }. Here we use state trajectories since different action spaces make it impossible to imitate actions. Since we put no assumption on the dynamics of the demonstrators, there is no guarantee that an optimal policy could be learned, e.g., in the extreme case, all the demonstrations could be -on-transferable and random policy will be learned. Our goal is to learn a policy with as high return as possible.
Since the dynamics of the demonstrators and the imitator are different, the transitions (sf, s442)) in the demonstrations may not be realizable by the imitator, i.e., no action aj in the imitator's action space A' could make (sf, af, ) > 0. Such demonstrations provide no useful information for the imitator and are non non-transferable. However, given such set of demonstrations from mixture of demonstrators, is non-trivial to directly test whether a trajectory achievable by reproducing the trajectory in the target environment since there is no action in demonstrations. Furthermore, different
3

---

demonstrators may take different policies due to different dynamics and may take different actions even the same state', This multimodal distribution of the demonstrations makes prior works on transferability measurement ineffective. In this paper, we aim to remove the interference from the multimodal distribution of the demonstrations and learn transferability to mitigate the negative impact of non-transferable demonstrations.
3.1 Sequence-based Contrastive Clustering
Since directly imitating from mixture of demonstrations will make unimodal policy suffer, need to fully capture and decompose the intrinsic multimodal distribution of data collected from different demonstrators. Thus, cluster demonstrations before learning their transferability to make sure they belong to a unimodal distribution. Motivated by contrastive learning [33], we learn sequence feature extractor and distance metric in the sequence feature space by contrastive learning over parefull-designed positive and negative pairs. We construct positive pairs by randomly subsampling fixed-length sub-trajectories from the same demonstration and treat sub-trajectories of different demonstrations as negative pairs. Such a design satisfies our requirement since the positive pairs are from the same trajectory, and thus are from the same mode. At each batch, we first sample N trajectories from E and for each trajectory &,. we take two sub- uub-trajectories <2t and esuh We then derive the contrastive learning loss for this batch of 2.V sub-trajectories follows:
N log Σ 2 exp( (F F(E)"))) + exp( (F F(C2)
exp((F((Eu)) 1), F(E
Conttast
N
where F is the feature extractor modeled as a recurrent neural network and (,, +) indicates the cosine distance. The loss maximizes the cosine similarity of features between positive pairs.
We embed contrastive learning into clustering to simultaneously ensure that the positive pairs are close and the clustering structure is learned. We first initialize matrix C of size dim(F) x K, where dim (F) is the dimension of the output of F and each column e₂ of C represents the center of cluster k. with K clusters in total. For each input sub-trajectory Sn we assign one-hot cluster
label yn E (0. as follows:
1, arg migminn=1.... (((n) ck|12 Vn.j
0,
otherwise
where assign the cluster label with the nearest cluster center sub-trajectory. Then we introduce the clustering objective with the contrastive learning loss Leentrast embedded into it:
Letuster = Coontrast + 2
(1 - Cyn
(3)
Every time we update the feature extractor F, we reassign the cluster label of each sub-trajectory by Eqn. (2) and update the cluster center by the following process:
Ck + 3)--8cck + Σ Ly 1)F(C
(4)
where II is indicator taking only when the condition is satisfied and 0 otherwise, and Ens, [=]] automatically controls the learning rate. We simultaneously optimize the contrastive learning objective and the clustering objective to encourage them to benefit from each other, where the cluster structure is learned to keep trajectories from the same mode to be in the same cluster. Our contrastive clustering algorithm derives K clusters of trajectories E E where E, contains trajectories with cluster label k. We show the algorithm procedure in Appendix.
3.2 Adversarial Transferability Measurement
Given that the sequence-based contrastive clustering algorithm removes the mutual interference of demonstrations from different modes, we then learn the transferability within each cluster. We
"This work mainly focuses on the case where the dynamics are nearly-deterministic like robotic applications unimodal policies can rarely realize multimodal demonstrations.

---

Sequence-hssed Clustering
Adversarial Learning
Trunsferabllity Inference
suboapping
Caub
G1
(s,s')
G.s')~EB
f -
DR
DE
CK
O/O
Transferability (5,5))
Cluster
Centers
O Domo
Imitator
Positive pair
Negative
Clustering
Figure 2: The outline of our whole algorithm can be divided into two phases. The first phase is sequence-bar based contrastive clustering where we simultaneously conduct contrastive learning and clustering. We create positive pairs by subsampling different sub-trajectories from the same trajectory and use sub-trajectories from different trajectories as negative pairs. The second phase learning transferability where we conduct an adversarial-leant -eearninnill based algorithm in each cluster.
design new transferability measurement by an adversarial-leaming algorithm based on generative adversarial imitation learning (GAIL) [5]. Our key insight is that the GAIL discriminator output indicates the likelihood for state transition to either come from the demonstration distribution or policy distribution. We train a GAIL policy my and a GAIL discriminator D& for each cluster of trajectories. The loss is defined as follows:
K
Liam Σ (B log(1 - Ds s1+1)) + ( (771))m log s s))+ (5)
Here Tr indicates the current policy in training and Dy indicates the discriminator. We use label 0 for the state transitions the demonstrations and 1 for the state transitions collected from the policy.
After convergence, the discriminator for each mode outputs a value in [o, 1] reflecting how likely an input transition is drawn from the state transition distribution derived by the imitator's policy. Thus for state transition in the demonstrations, if it has a discriminator output close to 1, it is more possible to drawn from the imitator's policy and achievable by the imitator. Thus, we quantify the transferability for state transition as follows:
Σ I((4) Ex] De(of)
(6)
Note that the discriminator is strengthened by the adversarial training paradigm and thus has a strong ability to discriminate whether a transition from the demonstrations the imitator's 's policy. The optimization of the discriminator is demonstrated to be efficient and effective [5], which requires no time-consuming step-by-step optimization in f-MDP [6]. Transferablity Imitation Learning. We finally embed the transferability quantified by Eqn. (6) into imitation learning. We normalize the transferability of all the state transitions into sampling distribution, where state transitions with larger transferability will be sampled more often:
Pu s
Σ Eeeg Lf JEE w( 88411
Using the fixed sampling distribution Pw. we can embed our transfcrability into any imitation- learning-frood frrom-bseervation algorithm [10, 5, 14]. We use GAIL [5] here to train the final policy:
LGAH. -Beoy (f )mp.. log(1 D( s(41) Eeo log(D , s441))
(8)
5

---

4 Experiments
We experiment with three MuloCo environments, a simulated Driving environment, and a simulated Franka Panda Arm environment. We compare approach with standard imitation learning algorithm: GAIL [5], imitation learning with a measure of feasibility: ID [25] and f-MDP [6]. The original ID learns the inverse dynamics model from random trajectories far from demonstrations and thus makes the learned inverse dynamics not work well on demonstrations. We create an advanced ID baseline by learning a GAIL policy generate trajectories as the data for inverse dynamic learning. Such trajectories are closer to demonstrations and make the inverse dynamics work better on demonstrations. We call the original ID ID-Random and the advanced ID as ID-GAIL.
We further conduct analyses including an ablation study to verify the efficacy of each component and include visualization of the learned transferability, the results for different compositions of demonstrations, and the performance gain when we are given a larger budget to collect demonstrations in the Appendix. Code is available at https: : //gi thub. com com/EvieQ01/00DII
4.1 MuJoCo
Figure 3: The illustration of the varying dynamics for HalfCheetah, Hopper, and Walker2d.
Environments. We illustrate the environments in Figure 3. The HalfCheetah is agent with two legs and a body. We create different dynamics by discounting the force of the front leg and the back leg with two factors 7f and "/6 respectively. We create a mixture of demonstrators by setting (11170) 76) as (i) (1, 0.9). (ii) (0.9, 1), (iii) (1, 0.05), (iv) (0.05, 1). The imitator has the original force: (7F+7)) (1, 1). The Hopper is an agent with one leg consisting of 3 joints. We create different dynamics by varying the gravitational constant. We create a mixture of demonstrators by setting the gravitational constant as (i) 15.0, (ii) 9.8, (iii) 2.0, (iv) 1.0. For the imitator, we constant of 12.0. The 2alker2d is an agent with two legs where each leg consists of 3 joints. We create different dynamics by using different frictions for the feet, i.e., the link that touches the ground, We create a mixture of demonstrators by setting the friction as (i) 24.8, (ii) 9.9, (iii) 3.9, (iv) 1.1. For the imitator, we use a friction of 19.9. For all three MuJoCo environments, we follow prior works [6, 25] to collect fewer demonstrations from more transferable environments. We collect 100, 100, 250, 250 demonstrations respectively for four dynamics in HalfCheetah, 10, 10, 250, 250 for Hopper and 25, 50, 50, 50 for Walker2d with 1000 interaction steps per demonstration. We train expert agents with Trust Region Policy Optimization (TRPO) [34] to generate demonstrations.
gravitational
Results. We show the expected return w.r.t. the number of interaction steps for the three environments in Figure 4. In all three environments, observe that the proposed method achieves the highest return. The baselines even show lower performance than naive GAIL under such multimodal distribution of source demonstrations because f-MDP learns a unimodal policy from the multimodal trajectory distribution, which cannot realize the demonstration trajectories, while ID fails to learn an accurate inverse dynamics model from random trajectories. The results demonstrate the importance of clustering trajectories of the same mode. By learning the transfcrability in each cluster, we can accurately filter out transrresfeee demonstrations and learn from transferable demonstrations.
4.2 Driving
Environment. In the driving environment, we can easily decide and interpret whether the target car can reproduce the route in the demonstrations. As shown in Figure 5(a), we create task where car drives starting from anywhere at the bottom side and ends at the top side. Two obstacles are set with the center at 1 width and 3 width respectively. We create different dynamics by setting obstacles with different widths and setting different speeds for the car, which simulates realistic scenario where different car models are driving at different places. The reward function is defined as 11 for each interaction step, + 1000 for reaching the goal, and 1000 for hitting the obstacle. In this environment, the different lengths and paths of the demonstration trajectories introduce clear

---

Ours
F-DPP
ID-anddom
ID-GAIL
Naive GAIL
3000
3000
2000
Reom
Return
2000
Retum
1500
1000
1000
1000
500
15
20
25
10
15
10
20
25
Steps
Interaction Steps (x10%)
Stops (x10%)
(a) HalfChectah
(b) Hopper
(c) Walker2d
Figure The imitation learning performance of different methods the three MuJoCo environments.
multimodal distribution, which can demonstrate the importance of clustering trajectories into an accurate mode. We create three demonstrators by setting the obstacles width as 10.1, [0.5, 0.25] and [0.25, 0.25) and setting the speed as 1.0, 1.0 and 5.0 respectively. For the target environment, we set the obstacle width as [0.4, 0.25] and the speed as 1.0. We collect 10", 2.5 x 10", and 2.5 10t interaction steps of demonstrations from each source dynamics, by handmade rules.
-250
Return
-500
-750
-1000
Navv GAILL
-1250
FMPP
20
40
60
80
Interaction Steps (x10f)
(a) Illustration
(b) Result
Figure 5: (a) The illustration of the driving environment, showing expert trajectories in multiple source dynamics and the target environment. (b) The results in the driving environment.
Results. As is shown in Figure 5(b), we observe that the proposed method outperforms all other baselines. Though ID performs well at the first few interaction steps but the performance drops much then, which can explained by that the feasibility learned by ID could find good demonstrations in the first few batches of data but makes errors then. That means ID can only learn partially correct feasibility. Instead, our method learns accurate transferability to filter out con-transferable demonstrations and converges stably high return. f-MDP, as the state-of-the-arr method for filtering non-transferable demonstrations, could not work well on this multimodal distribution of demonstrations, further indicating the importance of clustering trajectories into the correct mode.
4.3 Simulated Franka Panda
The environment simulates the Franka Panda Robot Arm² with 7 degrees of freedom (DoF), which implemented the PyBullet [35]. We create a task of pushing box from one side of the desk to the other. With the box set at base position and the robot arm set at random position with Gaussian distribution, we set the target at the right of the desk. We create different dynamics by disabling different joints of the Robot arm. As shown in Figure 6(a). create three demonstrators by disabling the No. 1, 4, and 6 joint respectively while disabling the No. 1 and 3 joints for the target imitator. The environment aims to address real problem to leverage historical data any robot to learn a policy for a new robot. Similar to the MuJoCo environment, we import more demonstrations from the more dissimilar demonstrator, where the number of interaction steps 1 10 x 10S and 1 x 106 for the environment disabling No. 1, 4, and 6 respectively. The reward function is defined as the current distance to the starting point, and an extra + +1000 for the box reaching the other side of the table and -1000 for the box dropping to the ground or the robot going past the box to the other side of the desk. We make demonstrations by handmade rules.
Environment.
Results. The expected return w.r.t. the number of interaction steps is shown in Figure 6(b). Our proposed method outperforms all the baselines by large margin. Directly applying GAIL introduces
"https//www.frankk..

---

700
sourre
500
Return
300
spuiree
100
-300
source
1
3
Interaction Steps (x10")
(a) Illustration
(b) Result
Figure 6: (a) The illustration of Simulated Franka Panda environment, where create different demonstrators by disabling different single joints and disabling two joints for the imitator. (b) The return of different methods in the Simulated Franka Panda environment.
low performance, which indicates that the demonstrations from other robots cannot be directly utilized learn new robot. The experiments show that the proposed method can serve as data cleaning step to clean the dataset from other robots for a real-robot transfer learning problem.
4.4 Analysis
Ablation Study. To verify that both our contrastive clustering algorithm and the adversarial-lcarning based algorithm con- tribute to the final performance, of our method with its variants by removing the clustering step and learning the transferability directly from the whole set of demonstrations (Ours w/o Cluster), and removing both clus- tering and the transferability (Ours w/o Cluster, Tran), which directly imitates the whole set demonstrations. We conduct these experiments in the Driving environment.
compare the performance
400
1000
Ours wil Claster
1200
1400
20
70
Steps (x105)
Figure 7: The ablation study two variants of our method.
The results are shown in Figure 7. We observe that Ours out- performs Ours w/o Cluster, which demonstrates that clustering trajectories within the same mode is important and our contrastive clustering algorithm achieves this goal. Ours w/o Cluster outperforms Ours w/o Cluster, Tran, which demonstrates that transferability is important to filter out non-tran con-transferable demonstrations and learn from more transferable ones.
Conclusion
We propose a new approach address oft-of-dynamics imitation learning (OOD-IL). Noticing that the demonstrations exhibit a multimodal distribution, we propose a sequence- contrastive clustering algorithm make trajectories from the same mode fall into the same cluster. We then propose an adversarial--carning based algorithm to learn the transferability of each cluster with the diseriminator output. Experimental results on three MuJoCo environments, a driving environment, and simulated robot environment show that the proposed method can learn a transferability measure accurately filter out non- transferable demonstrations and learn from more transferable ones.
Limitations and future work. While our work has substantially advanced OOD-IL, we believe that this problem merits further study. One primary limitation of current method is that method may not afford the computational cost of transferability learning when the number of clusters increases to hundreds or thousands. Pre-training or meta- vreaattaaning [36] before transferability learning may help boost learning efficiency, which is left for future work. A limitation of deploying our method into real system is that may not respect safety constraints due exploration during both transferability learning and imitation learning. Although this challenge is common for real systems, out-of-de out-of-dynamics learning may exacerbate and safe policy learning techniques can be adopted. Another promising future direction is to exploit a wider variety of demonstrations, thus enabling large-scale imitation learning. Although our work relaxes stringent assumption of identical dynamics, it is also limited since we assumed that collected demonstrations are optimal in the target environment, while it might not hold true in varying dynamics. In the future, we plan to address this combined challenge of multimodal OOD-IL and learning from sub-optimal demonstrations [37].

---

Acknowledgments
We would like acknowledge the support our wonderful coworkers: Yang Shu, Baixu Chen, Haixu Wu, and Yipeng Huang, whose generous help is an integral part of this work. This work was supported by the National Key Research and Development Plan (2020AAA0109201), National Natural Science Foundation of China (62022050 and 6202 1002), Beijing Nova Program (Z201 100006820041), and BNRist Innovation Fund (BNR2021RC00002)
References
[1] Zhang, Z. McCarthy, O. Jow. Lee, X. Chen, K. Goldberg, and P. Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In ICRA, 2018.
[2] F. Codevilla, M. Müller, A. López, V. Koltun, and A. Dosovitskiy, End-to-end driving via conditional imitation learning. In JCRA, 2018.
[3] S. Ross, G. Gordon, and D. Bagnell. reduction imitation learning and structured prediction no-regret online learning. In AISTATS, 2011.
[4] B. Ziebart, L. Maas, Bagnell, and A. Dey. Maximum entropy inverse reinforcement learning. In AAAI, 2008.
[5] J. Ho and S. Ermon. Generative adversarial imitation learning. In NeurIPS, 2016.
[6] Z. Cao, Y. Hao, M. Li, and D. Sadigh. Learning feasibility imitate demonstrators with different dynamics. In CoRL, 2021.
[7] M. Bain and C. Sammut. A framework for behavioural cloning. In Machine Intelligence 15, 1995.
[8] H. Daumé, J. Langford, and D. Marcu. Search-based structured prediction. Machine learning. 2009.
[9] S. Ross and D. Bagnell. Efficient reductions for imitation learning. In AISTATS, 2010.
[10] Torabi, G. Warnell, and Stone. Behavioral cloning from observation. IJCAI, 2018.
[11] Abbeel and Y. Ng. Apprenticeship learning via inverse reinforcement learning. In ICMI.,
2004.
[12] A. Y. Ng. J. Russell, et al. Algorithms for inverse reinforcement learning. In ICML, 2000.
[13] C. Finn, Levine, and P. Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In ICML, 2016.
[14] J. Fu, K. Luo, and S. Levine. Learning robust rewards with adverserial inverse reinforcement learning. In ICLR, 2018
[15] Y. Schroecker and C. L. Isbell. State aware imitation learning. In NewrlPS, 2017.
[16] F. Torabi, G. Warnell, and P. Stone. Generative adversarial imitation from observation. In ICML Workshop on Imitation, Intent, and Interaction (I3), 2019.
[17] W. Sun, A. Vemula, B. Boots, and D. Bagnell. Provably efficient imitation learning from observation alone. In ICML, 2019.
[18] C. L. Nehaniy, K. Dautenhahn, et al. The correspondence problem. Imitation in animals and artifacts, 41, 2002.
[19] P. Englert, A. Paraschos, J. Peters, and M. P. Deisenroth. Addressing the correspondence problem by model-based imitation learning. In ICRA Workshop on Autonomous Learning, 2013.
[20] Calinon, F. Guenter, and A. Billard. On learning, representing, and generalizing a task in humanoid robot. IEEE Transactions Systems, Man, and Cybernetics, 2007.

---

[21] C. Eppner, J. Sturm, M. Bennewitz, C. Stachniss, and W. Burgard. Imitation learning with generalized task descriptions. In ICRA, 2009.
[22] Q: Zhang, T. Xiao, A. Efros, Pinto, and X. Wang. Learning cross-domain correspondence for control with dynamics eycle-consistency. In ICLR, 2021.
[23] Z. Wang, Z. Cao, Y. Hao, and D. Sadigh. Weakly supervised correspondence learning. arXiv preprint arXiv:2203.00904 2022.
[24] Liu, Z. Ling, T. Mu, and H. Su. State alignment-based imitation learning. In ICIR, 2019.
[25] Z. Cao and D. Sadigh. Learning from imperfect demonstrations from agents with varying dynamics. IEEE Robotics and Automation Letters (RA-L), 2021.
[26] Sharma, L. Mohan, L. Pinto, and A. Gupta, Multiple interactions made easy (mime): Large scale demonstrations data for imitation. In CoRL, 2018.
[27] T. Gangwani and J. Peng. State-only imitation with transition dynamics mismatch. In ICLR, 2020.
[28] I. Radosavovic, X. Wang, I.. Pinto, and J. Malik. State-only imitation learning for dexterous manipulation. arXiv preprint 2004::20004046500 2020.
[29] M. Caron, Bojanowski, Joulin, and M. Douze. Deep clustering for unsupervised learning of visual features. In ECCV, 2018.
[30] H. Zhong, C. Chen, Z. Jin, and X.-S. Hua. Deep robust clustering by contrastive learning. arXiv preprint arXiv:2008. 03030, 2020.
[31] Y. Li, P. Hu, Z. Liu, D. Peng. T. Zhou, and X. Peng. Contrastive clustering. AAAI, 2021.
[32] D. Zhang, F. Nan, X. Wei, S. Li, H. Zhu, K. McKeown, R. Nallapati, A. Arnold, and B. Xiang. Supporting clustering with contrastive learning. In NAACL, 2021.
[33] Chen, S. Korbllihh, M. Norouzi, and G. Hinton, simple framework for contrastive learning of visual representations. In ICML, 2020.
[34] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. ICML, 2015.
[35] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. 2016.
[36] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. JCMI., 2017.
[37] Y.-H. Wu, N. Charoenphakdee, H. Bao, V. Tangkaratt, and M. Sugiyama. Imitation learning from imperfect demonstration. In /CML, 2019.
10


Reviews:
[{"review_id": "_6N6VZumYWqC", "reviewer": "Reviewer_Zz2p", "summary": "This paper focuses on recognizing multi-modalities in the learning from demonstrations, where the demonstrations come from different demonstrators, system dynamics, environments, etc. Building a demonstration clustering and transferability measuring procedures on top of the imitation learning allows the model to evaluate the similarity of the current and the demonstrated states. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros: \n\n- The motivation of the paper is well explained, i.e. help distinguish demonstrations from different dynamics and therefore allows the imitator to imitate the demos sharing similar states transitions without interference. \n- The three parts of the model, i.e. clustering demos, measuring transfer ability, and training the imitation learning model with transferability are well presented.\n- The paper has a proper ablation study and further Hyper-parameter tuning details in the appendix.\n\n\nCons:\n- It is a pity that no real-robot experiment is conducted, which decreases my confidence of mine that this work can be easily deployed in a real system. Maybe the authors can share some knowledge or experience about the limitations of deploying it in a real system. \n- The experiment details of the third task is a little bit unclear to me, i.e., what is this \"moving one box from one side to the other\"? The task description and Fig.6 do not give me enough information about the experiment, and neither does the supplementary material. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "In general, I think this paper reaches the minimum threshold of CORL, as it is well-written, especially in the model structure part. The reasons that I do not feel very confident to recommend this paper are (1) I do not have strong background knowledge of the methods used in this paper or related works and (2) there is no real robot experiment, which is not common in imitation learning. I am looking forward to getting some feedback from the authors regarding the reasons for not deploying current work in a real system and the scalability of the number of demonstrators.  ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "- A question in my mind, what is the reason for not using a mixture model, e.g. GMM, to model this multi-modality demonstrations behavior?\n- I like Figure 2 in the appendix, but I would like to know how the authors compute the transferability of the other 3 methods.\n- As I said in the weakness section, what is the bottleneck of getting a real robot experiment? From my understanding, a task like \"driving\" should be able to be achieved by a robot in task space. In general, it should be easier for imitation learning to work with a real system than RL, especially because the authors presented a lot of real system settings in the introduction section. \n- Where did the demonstrations come from? From RL agent or some human input or hard-coded trajectories?\n- The number of clusters, K, seems crucial in the current model. So, what is the relationship between K and the number of demonstrators? Imagine a practical usage case, like autonomous driving, with different cars, weather, and road conditions, even rules (driving on the left in the UK vs. on the right in the US), it seems that K will be very big. So what is the scalability of the current model?\n"}}, {"review_id": "J6cl6MkkQnm", "reviewer": "Reviewer_UJgv", "summary": "This paper proposed a novel imitation learning method that utilizes multimodal demonstrations from different dynamics and action spaces.\nThe main contribution of the proposed method is to develop an algorithm that relaxes the assumptions in the general imitation learning and explicitly takes into account the multimodal nature of demonstrations, which has been overlooked in previous studies.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Under a more general problem setting, this paper succeeded in imitating from the multimodal demonstrations without simlifying the problem.\n- The proposed method is able to absorb differences not only in dynamics but also in the action space.\n- The performance of the proposed method was properly verified by benchmarks, including ablation studies.\n\nWeaknesses:\n- When dealing with multimodality, it is inconvenient to design the number of clusters K to be specified in advance.\nIn particular, since the proposed method requires training pi_k for each cluster, it is not desirable to guarantee generality by making K too large without consideration, but too small K cannot capture the multimodality.\n- I think the two-stage GAIL should eventually make the learning process highly unstable.\n- The final pi to be acquired is obtained by GAIL using a subset of the multimodal dataset that is close to the imitator's dynamics, which should inherit multimodality.\nIn that case, depending on the probability model of pi, it may fail to capture the multimodality and underfitting may occur.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The problem setting that this paper challenges is very important and versatile from the viewpoint of practicality, and the proposed method steadily resolved the issues that arise in this, although its approach seems to be heuristically constructed.\nThe performance of the proposed method was properly verified, and the effectiveness of the method seems to be confirmed.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "I would like to see additional ablation studies for checking the effect of the number of clusters.\nIn particular, please clarify the limitation in terms of learning efficiency and computational cost.\nIn addition, we would like to know if the Interaction Steps on the horizontal axis of the learning curve in the paper include those required for learning pi_k.\nIf not, the authors should clarify the number of iteration steps required for learning pi_k, which reveals the decrease of learning efficiency in total.\n\nFrom the benchmarks provided, it seems that the sub dataset extracted according to the transferability measurement is designed to avoid multimodality.\nIt would be better to evaluate the performance when it is multimodal, if possible.\n\nOnly for defining transferability measurement, it may not be necessary to cluster the dataset, and it seems sufficient to construct a metric space.\nIt would be good if the authors could add the necessity of clustering in this regard."}}, {"review_id": "Cao2vrwsmbQ", "reviewer": "Reviewer_sqom", "summary": "This paper presents a new Imitation Learning method designed to address action-space mismatch between demonstrator and learner. The action-space mismatch assumption manifests in the assumption that no expert actions are available, and also implies that the underlying dynamics of the demonstrations $p(s' | s, a_{\\text{expert}})$ does not match the dynamics of the learner $p(s'|s, a_{\\text{learner}})$. This is termed 'OOD-IL' (Out-of-dynamics IL). The proposed method uses contrastive learning and clustering to learn a set of discriminators and policies in a GAIL framework, which is designed to address potential multimodality in the training data. The resulting set of discriminators are used to form a joint distribution over state transitions that is intended to capture whether a particular state transition is feasible. This joint distribution is then used as the sampling distribution for performing a final stage of imitation learning in an 'OOD' (out-of-dynamics) environment.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- The OOD-IL regmie is certainly of interest, as successful methods in this of relaxation of typical imitation learning would enable a much broader set of training data to potentially be used for imitation learning. \n- The proposed method appears to work better than prior work in this regime. \n- Ablations confirm the efficacy of the clustering component and the transfer component on one environment\n- A sweep of the clustering hyperparameter (in the appendix) further confirms the efficacy of the clustering component\n\n### Weaknesses\n- W1 (main weakness): The main experimental flaw is that there seems to no evidence presented in support of the claim that the multimodality of the demonstrations is the main factor behind the performance difference between the proposed approach and other approaches. This is an important weakness because it is part of the paper's central claims. E.g. L48-50 \"The multimodal demonstrations make it impossible to learn a deterministic policy from the f-MDP. Furthermore, f-MDP suffers from slow and inaccurate optimization difficulties due to a step-by-step optimization procedure\"; L124-125 \"This multimodal distribution of the demonstrations makes prior works ineffective. In this paper, we simultaneously remove the interference from the multimodal distribution and learn transferability to mitigate negative impact of non-transferable demonstrations.\" I expand on W1 below.\n\n- W2: Discussion of prior methods that also perform clustering by contrastive learning are missing. I suppose the idea is natural, but the current paper would be strengthened by drawing connections to this line of prior work and discussing their significance to the current work. These papers look related [A, B, C].\n\n- W3: I doubt the following claim. L181 claims that \"we can embed our transferability into any imitation learning algorithm.\" More explanation is needed for why this is the case. For instance, how could one use this sampling distribution to perform BC? The expert actions at samples from $p_w$ are not necessarily available. Therefore, BC is a counterexample to refute this claim on L181. Either the claim needs to be removed or refined (inclusion of a refined explanation and/or evidence). The regime studied in the paper is particuarly one in which the expert actions for the robot are not available, so it seems to rule out any imitation learning algorithm that relies on these (i.e. we can only use state-based imitation learning algorithms). If that's the conclusion, it would be good to include in the appendix (and reference in the main text) an example objective function of a different imitation learning algorithm making use of $p_w$.\n\n- W4: The limitations section needs work. It doesn't actually discuss limitations of the proposed method in the experimental settings investigated, which suggests that it 'solves' them.\n\n**More details on W1**:\nBecause there are other differences between the proposed approach and prior works, a direct comparison between them does not necessarily identify the multimodality of the demonstrations as the source of the performance difference. Therfore, Figs 4, 5b, and 6b, -- the main analyis -- do not constitute conclusive evidence in support of the claim. Fig 7 contains an ablation of the clustering component, and the clustering component allows for more policies to be learned to fit the original (pre-transfer) data. However, even with the clustering component removed, because the policies are stochastic, they still have the capacity to fit multimodal trajectory distributions. Therefore, to me it seems there is no evidence presented in support of the main claim.\n\nHere are my suggestions for experiments to run to collect this evidence: \n- (1) Re-run the ablations in Fig 7 and also include ablations of the proposed method for the other environments, except with deterministic policies. This will force each policy to be unimodal, and therefore *only* the clustering (inclusion of multiple policies) would allow for fitting of multimodal data. Comparing the results of the four variants: {with clustering, without clustering} X {with deterministic policy, with stochastic policy} is the key idea here.\n- (2) Confirm that the performance benefits of fitting multi-modal data go away when the data is made unimodal. This could be done by modifying the Driving environment so that only unimodal demonstrations and task. Then, there should be little-to-no performance difference between the ablations.\n\n#### Minor weaknesses\n- MW1: S4.4 and Fig7 are unclear -- what is the environment used for these experiments? I'm guessing it's the driving environment (based on the similarities of the y-axes in Fig5b and Fig7), but this needs to be made unambiguous.\n- MW2: L202 \"Hooper\" -> Hopper\n\n### References\n- [A] \"Deep Robust Clustering by Contrastive Learning\" https://arxiv.org/pdf/2008.03030.pdf\n- [B] \"Contrastive Clustering\" https://ojs.aaai.org/index.php/AAAI/article/download/17037/16844\n- [C] \"Supporting Clustering with Contrastive Learning\" https://arxiv.org/pdf/2103.12953.pdf\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The regime is interesting, and the method is interesting, yet the central claim is lacking evidence (see W1). While I think the work has potential for impact, this lacking evidence makes the paper technically lacking, and therefore I do not recommend acceptance of the paper in its current state.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "See the weaknesses section, in particular, W1. Based on my current understanding, following the protocol suggested in the experiments for W1 would be sufficient to address W1, if it indeed turns out to confirm that multimodality is the important key factor. However, it could turn out to be the case that these experiments do not show the multimodality to be important, in which case, I think the results would necessitate further explanation and the paper significantly reworked around a different hypothesis that is consistent with the experiments."}}, {"review_id": "B9pzRe1bMJ3", "reviewer": "Reviewer_M2Dq", "summary": "The paper considers the problem of imitation learning from out-of-dynamics demonstrations. Since the demonstrators may have different dynamics than the imitator, these demonstrations may not be useful to the learner, if not detrimental. The paper follows existing work and uses a 2-stage procedure. It first learns a feasibility/transferability score for the demonstrations, and then uses an imitation learning algorithm on weighted data. The paper addresses a major limitation of f-MDP, the state-of-the-art approach, which is not able to deal with multimodal demonstrations. The proposed approach first clusters the demonstrations, then learns a policy for each cluster. Additionally, it uses a transferability metric that is easier to compute than f-MDP. The proposed approach is evaluated on various simulated tasks with different physics backends such as mujoco and pybullet.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- **Motivation.** The research problem is well motivated. The challenges of the research problem, as well as the limitations of existing methods are presented clearly.\n- **Organization.** The paper is in general well organized. The technical approach is easy to follow, and seems technically sound. \n- **Literature survey.** The authors did a thorough review of related literature. The authors clearly present the limitations of existing work, and how the proposed work is able to address these limitations.\n- **Ablation studies.** The authors present a series of ablation studies verifying the importance of each component, as well as demonstrating how each hyper-parameter is chosen. \n\n### Weaknesses\n- **Lack of real-robot results.** Although the proposed method is evaluated on different physics engines such as mujoco and bullet, it lacks experiments on real hardware. It seems like there are still considerable gaps between the simulation and real hardware experiments. First of all, the physics of real hardware could be very different from simulation. For example, in the videos on the website, the Franka robot and the box are often “sinking” into the table. Second, it is possible that demonstrations generated by RL policies in simulation differ from demonstrations that can be collected on the real hardware. Therefore, it would be nice if the authors can present results on real hardware, show that real-world demonstrations also have such multi-modal properties, and demonstrate that the proposed method outperforms baselines for such data. \n- **Comparison with other clustering algorithms.** It would be nice if the authors can compare their results with simpler clustering algorithms, such as K-means. Although in the appendix, the authors claimed that L2 distance can’t be used on trajectories of different lengths, it seems like trajectory segments with fixed length are actually used according to the appendix.\n- **Loss function.** In Equation (8), it seems like $p_w$ is also a function of the discriminator $D$. How did the authors compute the gradient of $D$ in this case? If the authors did not consider the gradient of $D$ from $p_w$, the authors may want to note that in the paper.\n- **Final policy.** It seems like $K$ policies are trained by the proposed algorithm. Is the reported results from the mixture of $\\pi_k$ or the best one?\n- **Source of multimodal demonstrations.** It seems like that in a lot of the experiment setup, the demonstrations are multimodal because they come from different dynamics. It would be nice if the authors can compare their results with a baseline where a policy is learned for each demonstrator (with transferability but without clustering). This can significantly strengthen the paper by verifying the importance of the clustering step. \n- **Data distribution for mujoco environments.** It seems that the composition of demonstrations is quite different among the mujoco environments. It is appreciated that the authors show results with different data composition for Hopper, but it would be nice if the authors show similar results for the other two environments. \n- **HalfCheetah.** It seems like for HalfCheetah, all demonstrations are feasible, since $\\gamma_f$ and $\\gamma_b$ from each demonstrator is smaller than the imitator. Is that true?\n- **Ablation study with trans.** It is appreciated that the author presented the ablation study. It would be nice if the authors also compare their results with the one that uses clustering but not the transferability step. It can help validate the importance of the transferability step.\n- **Common $\\rho_0$.** It seems like the authors assume that all of the MDPs share the same initial state distribution, why is such assumption necessary? By the way, It seems like $\\rho_0$ is actually defined in the MDP paragraph.\n- **Feature extractor F.** Feature extractor $F$ first appears in Equation (1), but is not described until line 149. The authors are also advised to provide more details on the structure of $F$ that was used in the experiments.\n- **Typo?** In Equation (5), should it be $(s_t^d, s_{t+1}^d)\\sim\\Xi_k$ under the first expectation? \n- **Transferable demonstrations.** In appendix C.3, the authors mentioned “equal number of transferable and non-transferable demonstrations”. How do the authors know which demonstrations are transferable, which are not?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper addresses an interesting research problem that is well-motivated. The authors were able to clearly explain the contribution of the proposed work. Although the proposed method seems technically sound, the contribution seems a little incremental. The main weakness of the paper comes from the experiment section. I am concerned about the paper because no real robot experiments are presented, and some additional results are needed to support the claims. \n\nWith that said, I am happy to increase my score if the authors present more experimental results during the rebuttal period. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See **Strengths and Weaknesses**.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)