Learning Feasibility to Imitate Demonstrators with Different Dynamics
Authors: Zhangjie Cao
Yilun Hao
Mengxi Li
Dorsa Sadigh
Venue: CoRL2021 Poster (2021)
Keywords: Imitation Learning
Learning from Agents with Different Dynamics

Abstract: The goal of learning from demonstrations is to learn a policy for an agent (imitator) by mimicking the behavior in the demonstrations. 
Prior works on learning from demonstrations assume that the demonstrations are collected by a demonstrator that has the same dynamics as the imitator. However, in many real-world applications, this assumption is limiting --- to improve the problem of lack of data in robotics, we would like to be able to leverage demonstrations collected from agents with different dynamics. This can be challenging as the demonstrations might not even be feasible for the imitator. Our insight is that we can learn a feasibility metric that captures the likelihood of a demonstration being feasible by the imitator. We develop a feasibility MDP (f-MDP) and derive the feasibility score by learning an optimal policy in the f-MDP. Our proposed feasibility measure encourages the imitator to learn from more informative demonstrations, and disregard the far from feasible demonstrations. Our experiments on four simulated environments and on a real robot show that the policy learned with our approach achieves a higher expected return than prior works. We show the videos of the real robot arm experiments on our website.

Learning Feasibility to Imitate Demonstrators with Different Dynamics
Zhangjie Cao', Yilun Hao', Mengxi Li,, Dorsa Sadigh Department of Computer Sciences, Stanford University, United States *Department of Electrical Engineering, Stanford University, United States eas stanf ord. edu, Eyilunhao, mengxili) @stanfffrttttoddddooooooo edu, dorsaucs stanford edu
Abstract: The goal of learning from demonstrations is to learn a policy for an agent (imitator) by mimicking the behavior in the demonstrations. Prior works on learning from demonstrations assume that the demonstrations are collected by a demonstrator that has the same dynamics as the imitator. However, in many real-world applications, this assumption is limiting - to improve the problem of lack of data in robotics, we would like to be able leverage demonstrations collected from agents with different dynamics. This can be challenging as the demonstrations might not even be feasible for the imitator. Our insight is that we can learn feasibility metric that captures the likelihood of demonstration being feasible by the imitator. We develop a feasibility MDP (f-MDP) and derive the feasibility score by learning feasibility measure encourages the imitator to learn from more informative demon- strations, and disregard the far from feasible demonstrations. Our experiments on four simulated environments and on a real robot show that the policy learned with our approach achieves higher expected return than prior works. We show the videos of the real robot arm experiments on our website.
optimal policy in the f-MDP. Our proposed
Keywords: Imitation Learning, Learning from Agents with Different Dynamics
Introduction
Imitation learning aims to learn a well-performing policy from demonstrations. Standard imitation learning algorithms usually assume that the demonstrator (the agent that generates the demonstrations) and the imitator (the agent that is learning policy) share the same dynamics, i.e., the transition functions are the same [1, 2, 3, 4]. Specifically, in a given state, with the same action, both the demonstrator and the imitator transition to the same distribution of next states. However, this assumption limits the usage of already collected demonstrations. Imagine a setting, where a set of demonstrations are collected book on the empty area of the shelf (on the left) without colliding with the books that are already placed on the right side of the shelf. Later, we might decide to buy different arm with 3 DoF (e.g.. only the joints circled in green as shown in the figure are used). We would like to learn policy for this 3 DoF robot arm that can achieve the same task task-placing the book on the empty region of the sheif-using the originally collected demonstrations on the 7 DoF arm. In general, our goal is to enable using and reusing data collected on robots with different dynamics or embodiments tackle the problem of lack of in-domain data in robotics. The 3 DoF robot arm should still be able to learn a policy based on feasible or nearly feasible demonstrations from an agent with different dynamies, e.g., using the trajectories that go over the bookshelf in Fig. 1. Motivated by this example, we relax the assumption of shared dynamics between the imitator and demonstrator so that the data can be collected from demonstrators with the same state space but different dynamics from the imitator, e.g.. demonstrators with different embodiments, body schemas, joints, or rigid body structures.
7 Degrees of Freedom (DoF) robot arm shown in Fig.
place
Prior works in imitation learning from demonstrators with different dynamics typically rely on state-only demonstrations and learn a policy to maximally follow the sequence of states in demonstra- tions [5, 6]. Such learning techniques assume that all of the collected demonstrations are useful for the imitator. However, it possible that demonstrations drawn from agents with other dynamics can
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

(Ei))
Demonstrations
7-DoF Robot Arm (Demonstrator)
3-DoF Robot Arm (Imitator)
Figure I:
An example of imitating demonstrators with feasibility. The left image shows that a set of demonstra- tions (blue and red trajectories) are available for the 7 DoF rohot arm. We aim to learn a policy for the 3 DoF robot (joints are circled in green) by learning from the demonstrations of the 7 DoF robot (blue is feasible and red is infeasible). We learn a feasibility score reweight each demonstration conduct imitation learning.
be useless even harmful for the imitator because they may not be feasible for the imitator. Going back to the example in Fig. I, the red trajectories that move around the stack of books are not feasible for the 3 DoF robot arm. Imitating such trajectories may cause the 3 DoF robot arm maximally follow these trajectories and even collide with the existing stack of books. Therefore, it is crucial to identify and avoid trajectories that are far from feasible for the imitator, and instead learn more from useful demonstrations, e.g., the blue trajectories that go over the shelf that are still feasible for a robot with DoF.
To avoid the influence of useless or harmful demonstrations from agents with different dynamics, we rely on feasibility score, which measures how feasible trajectory is for the imitator, and select trajectories with high feasibility to imitate. For example, the blue trajectories should have higher feasibility than the red trajectories in Fig. 1. Prior work such Cao and Sadigh [7] estimate the feasibility score by computing the distances of demonstrations and corresponding trajectories but the performance highly relies the accuracy of the inverse dynamics model, which can be difficult to learn. Our key idea is to directly learn a feasibility score for the imitator based on the collected demonstrations. Specifically, a feasibility Markov Decision Process (f-MDP) based on the imitator's MDP and the trajectories provided by the demonstrator. The optimal policy for the f-MDP maximally follows the behavior of the demonstrations but is limited by the imitator's 's environment. This optimal policy helps assign a feasibility score over the demonstrations. We conduct imitation learning on the demonstrations re-eeighted by the feasibility score to learn the final policy for the imitator. We experiment with several simulation environments and a manipulation task with Panda Franka arm. We show that the policy learned from demonstrations re- re-weighted by our feasibility achieves higher performance compared to other methods.
model the imitator environment
MDP and build
2 Related Works
Imitation Learning. Imitation learning seeks a policy that best imitates demonstrations. Current imitation learning methods can be roughly divided into Behavior Cloning (BC), Inverse Reinforcement Learning (IRL) and Generative Adversarial Imitation Learning (GAIL). BC directly learns the policy from sequence of state-action pairs via supervised learning [8]. where dataset aggregation [9] or policy aggregation [10, 11] are proposed to address the compounding errors problem. IRL first Iearns reward function that best matches demonstrations and then finds policy through reinforcement learning to maximize the recovered reward [1, 12, 2, 13]. GAIL learns the expert policy by matching the occupancy measure between the policy and the demonstrations [4].
However, most imitation learning works require that the demonstrations consist of a sequence of states and actions. When only state observations are available, new imitation learning algorithms are proposed to address the lack of actions. Torabi et al. [14] recover the actions between consecutive states through an inverse dynamics model. GAIL-based works directly match the state occupancy measure between the demonstrations and the policy [15, 16, 17]. However, imitation learning methods learned from either aate-action state-only demonstrations assume that the demonstrator and the

---

imitator have the same dynamics. Since demonstrations from different dynamics may not be feasible for the imitator, directly imitating cannot achieve the same optimal behavior, and may cause unknown suboptimal outcomes. Thus, standard imitation learning algorithms do not fit our problem setting.
Learning from Demonstrations with Different Dynamics. Early works model this problem correspondence problem between the demonstrator and the imitator, and map states and actions in demonstrations the imitator's states and actions [18, 19]. Englert al. [20] align the state trajectory distributions to address the correspondence problem. Calinon et al. [21] model the demonstrations as a Gaussian mixture model within projected Nower-dimensional subspace. Eppner et al. [22] learn a task description. Domain randomization methods learn the correspondence as an invariant latent space by randomizing domains [23, 24, 25]. Zhang et al. [26] learns a translation mapping to model the correspondence. However, modeling correspondence requires that there exists a strict correspondence between the MDP of the demonstrator and the imitator. Recent works instead only assume the shared state space between the demonstrator and the imitator, and address the different dynamics problem by encouraging the imitator to maximally follow the state trajectory of the demonstrator [5, 27, 28, 6]. However, all these works ignore an important challenge- that is the demonstrations may be far from feasible for the imitator. Enforcing the imitator to follow such trajectories may lead to unknown behavior. We focus on this challenge and develop a feasibility score to down-weight demonstration trajectories that are far from feasible for the imitator. Compared to the works that learn feasibility to filter infeasible trajectories [7]. we do not require the inverse dynamics model, which can make our setting more generalizable to different environments.
3 Problem Statement
In our problem setting, an imitator aims to learn from demonstrations collected from N demonstrators with various dynamics. We formalize the demonstrators and the imitator each as a standard Markov decision process (MDP). For each demonstrator j. (1 S j I N). the MDP is formalized as 7). The MDP for the imitator is M = AS, p², R, Po. 7). S is the Mý = AS, pt, R, Po:
shared state space for all environments. Ad and A' are the action spaces and p.. S x AJ X [0. and p² S A x S + [0, 1] are the transition probabilities for each demonstrator and the imitator respectively. Note that in our problem setting, we use the transition function to denote dynamics and the demonstrators and the imitator may have different dynamics and action spaces. Pu is the shared initial state distribution for all MDPs. R: S x S + IR is the reward function. Note that we make the assumption that the reward function is based on state transitions and is shared between the demonstrators and the imitator, which is common assumption used in prior work [5, 7], and is usually satisfied since the demonstrators and the imitator conduct the same task in the same context. is the shared discount factor. A policy for the imitator x' S x A + [0, 1] defines probability distribution over the space of actions in a given state. An optimal policy TT maximizes the expected return 7 Es E 7 (RR(s,, de, 8t41)], 1)], where indicates the time step.
We aim to learn policy T' for the imitator, given set of demonstrations from different demonstrators where each trajectory is sequence of states { sf. sq.... sgy We assume that the optimal policy can be learned by imitating the useful demonstrations, which is a general assumption adopted by prior imitation learning works [8, 4, 5, 7]. The violation of this assumption, as shown in prior works, leads to learning a suboptimal policy. Note that we discard actions from the demonstrations instead of imitating the state-action trajectories because different action spaces between the demonstrators and the imitator make it impossible to imitate the actions.
E' = ".f.....EbbsEtt..
Challenges. The core challenges of imitation learning from demonstrations with different dynamics are: (1) How imitate useful demonstrations with different dynamics, (2) How to avoid harmful demonstrations misleading the imitator. Prior works have studied and made progress for the first challenge [5, 6]. but the second challenge is still under-studied. Strong assumptions such as access to or learning an accurate inverse dynamics model are needed to filter out harmful demonstrations [7]. We address the second challenge by learning a feasibility score that measures how likely for a demonstration to be feasible for the imitator with minimal assumptions: only using the environment of the imitator, i.e., we can collect interaction data in the environment but we do not know the exact reward and transition function of the imitator.
3

---

r (ot st T,
{of) - 50 st dg A' ()). - (b So = si
stt1
7-1
$1
One-step f-MDP (5, A' p''dp''pp.... Uniform(TT).r'' Eqn. (1)
To = (46)
50
S E A' Stes
p pl()
(6 Re St41) =-1
50 %0
st
83
Demonstrations
Trajectory E-DP (S A', pt ..ntform()).. Egn. (2)
The illustration and comparison of one-step f-MDP and trajectory f-MDP, The blue state transition and trajectory from the demonstrations while the orange state transition and trajectory rollouts in the f-MDP. One-step f-MDP collects the states the Former Set uses the uniform distribution over states as the initial state distribution. Trajectory f-MDP collects the initial state of all the demonstrations and uses a uniform distribution over them as the initial state distribution.
Figure 2:
4 Feasibility-Based Imitation Learning
The feasibility of a trajectory depends on the feasibility of each state transition in the trajectory, i.e. if (st,St+1) St+1) is feasible for all time steps. A state transition (s;, St+1) is feasible when there exists an action of E A' such that p' (st, oi, 81+1) 1 for deterministic transitions or p' (st, of, 8++1) > for stochastic transitions. In this section, we discuss the deterministic MDP setting and discuss the stochastic setting Appendix.
Feasibility can be directly measured by a perfect inverse dynamics model ʃ : S x S + A that takes state transition (8+,8)+1) E S x S as the input and outputs the action o, E A that achieves the transition if feasible or outputs 'Infcasible'. However, having access to this model is often non-trivial and such binary feasibility measurement as f discards all infeasible demonstrations without considering any useful information from slightly infeasible trajectories.
Our goal is learn policy n²' S > A'' for the imitator maximally achieve the state transitions in the demonstrations. This means that if the state transition (s sd s/+1) from a demonstration is feasible, the next state produced by x², i.e., s/41 = (s( (x{)) should be equal to 8441- Otherwise, we would like the policy to output an action that ensures the next state si+1 is as close as possible to the next state from the demonstration sf-11. Therefore, the distance between 8/.1 and of41 can serve as measure of feasibility, where a smaller distance corresponds to a higher likelihood feasibility. To learn this policy, we design a feasibility MDP (f-MDP), where we ensure that the optimal policy of
the f-MDP satisfies the above requirement. f-MDP is defined M A' A', p², RI, (2:.6''7) We will now discuss our choices for the components of f-MDP.
One-step f-MDP. First, recall that our goal is to learn policy for the imitator to maximally achieve the policy should be learned in an environment with the
the state fransitions in the demonstrations. same state-action space and transition probability as the imitator. We would like the reward of the f- MDP to encourage maximally achieving the state transitions in the demonstrations. Let us first collect all the state transitions T = {($ s(+)) } in all of the demonstrations. We define the Former Set to be the set of states in the demonstrations that
transition from: TF {sी (5 #1+) E T). The initial state distribution P6 can be defined uniformly over the Former Set as Uniform (Ty). Here, we assume that all the states in the Former Set can be visited by the imitator. We define the reward of One-step One-step.-MDD f-MD so that it matches the one-step transitions from the Former Set
(1) where (sdl sfi-)) is state transition in the demonstrations and a E A' is sampled from the action space of the f-MDP. fas is a function that measures the distance between the states (e.g.. the L2 distance). We define the reward to penalize the distance between s' and sil
og Uniform(TF), S == s' p'p'ss,)), R (s, s') ffss (s', s(+1),
Trajectory f-MDP. The one-step f-MDP suffers from an important shortcoming: the assumption that all states in the Former Set must be visited by the imitator can be violated, because the demonstrators have different dynamics from the imitator and some demonstration states can never be reached by the imitator. So we cannot set Uniform(T (T)) the initial state distribution for the f-MDP, We instead

---

collect the initial state si of all the demonstrations, To = { and define the initial state distribution of the Trajectory f-MDP as Uniform(To) Since all the demonstrators and the imitator share the initial state distribution, all states in To can be visited by the imitator. We define the reward as:
$ Uniform (To), Su sg:
$1-1 $1-1 = p²
as),a, p'),a)
(sL, a, $142) -fais ($ s6+1):
(2)
We use the L.2 distance for fais in our experiments. Similar to the one-step f-MDP a E A' is sampled from the action space of the imitator.
Learning Feasibility. Given the Trajectory f-MDP defined above, for each demonstration trajectory 5. the highest reward achieved in this f-MDP reflects the feasibility score of the trajectory. We use reinforcement learning to learn the optimal policy of the Trajectory f-MDP, ... We then derive the feasibility of each demonstration trajectory as a function of the trajectory f-MDP reward:
w({) = exp ( (=Emm(1))
( o5) -C
fain(se
(3)
St is the state at step & in the rollout derived by the policy x². We use an exponential function of the cumulative reward since the cumulative reward is always negative and the exponential function can bound the feasibility in the range of [0, 1]. The parameter C' is used to shift the function to avoid the situation where the cumulative reward is extremely negative, while the parameter a controls how low the reward can be, and when a demonstration can be fully filtered out by assigning a feasibility of close to O. In practice, C' is usually set as the maximal cumulative reward over all demonstrations to ensure the maximal feasibility is 1.
For the feasibility of each state transition (o s(+1), we use the feasibility of the trajectory it belongs to: w((sf, s1 #)+1) (E.), where (sf, sq & We do not the state distance at each time step between 1 and o6 in the One-step f-MDP because such measurement suffers from the fact that within a trajectory, the reward of later steps are influenced by former steps. For example, if st diverges from st. St+1 will diverge more from 4 So the per-step reward is an unfair measure of feasibility for the state transition (s", sd 1i+1) at different time steps f. Therefore, we use the accumulative reward of the whole trajectory as our feasibility measure, where all the state transitions share the same value.
The discount factor y is usually set as nd < 1 to reduce compounding errors. Specifically, the length of rollout in the f-MDP is the same as the corresponding demonstration, which can be very long. If the state in the rollout starts to diverge from the demonstration trajectory at f. meaning that ||st I| > 0, the steps after time step i even diverge more from the demonstration. This makes the trajectory reward for all the infeasible trajectories very low and does not allow for diseriminating among different infeasible trajectories. Therefore, we set discount factor of yd < 1 to discount or even ignore the trajectory reward at later steps.
Leveraging our Trajectory f-MDP design, feasible trajectories still receive the maximal reward of O since each state in the rollout will perfectly match the demonstration thus having a feasibility of 1 as in Eqn. (3). Instead, infeasible trajectories receive negative rewards leading to smaller feasibility scores, which reflects how far away the demonstration is from the closest feasible trajectory.
One may worry about the time complexity of our approach since we need an additional RL training to learn an optimal policy for the f-MDP. However, the f-MDP is lot simpler compared to the imitator's s MDP since the initial distribution is reduced from the distribution of all possible states in the demonstration set to discrete distribution over the initial states of the demonstrations. This can simplify the time complexity of finding the optimal policy for the f-MDPs.
Algorithm. Using the feasibility metric in Eqn. (3), we assign each state transition with the same feasibility of the trajectory it belongs to. Directly weighing the imitation loss as [29] may lead to gradients that are close to 0 if batch of data all have low feasibility. This can make the algorithm inefficient by wasting samples from many iterations. Instead, for a more efficient training, we define a discrete probability distribution P over the collection of state transitions in all the demonstrations: T. (o((s(,=ff)))
where the probability of a tatte-transitonn (s s Pw ((s) s(41)
State transitions with larger feasibility will be sampled more often. Using the sampling distribu- tion Pus. we can embed our method into any imitation learning algorithm to enable learning from demonstrations with different dynamics. We show the algorithm block in the Appendix.
5

---

(a) Swimmer Illustration
(b) Walker2d Illustraiion (c) HalfCheetah Illustra- (d) Hopper Illustration
tion
100
canne seturn
3000
Expected Retura
2000
2000
23
000
Nomber Internetion Stages (1x7)
Number Internction Steps (1=7)
Number Interatioon Staps (107)
Stande tettrmeetii Stape
(c) Swimmer Results
(f) Walker2d Results
HalfCheetah Results
(h) Hopper Results
Ours
ID-Feas
SAIL
GAIL
Figure 3: Illustration of different dynamics in Swimmer: varying the joint limit of the front and back joints (ay and as)) (b) Walker2d: varying the friction of the feet (8). (c) HalfCheetah: varying the joint control force of the front and back joints by multiplying a factor 7, and t with the front and back joint force. (d) Hopper: varying the gravitational constant respectively (e-h) show the expected return in these four environments.
Sampling More Demonstrations with the Feasibility Score. When the existing useful demonstra- tions are too scarce to learn a well-performing imitation learning policy, we need to acquire more demonstrations from the demonstrators. But collecting new demonstrations can be expensive, so often can only acquire a limited budget demonstrations. We thus need to collect the most useful demonstrations within this limited budget. The proposed feasibility metric provides criterion to decide the similarity between the imitator and each demonstrator. If demonstrator has higher similarity, we sample more from this demonstrator because its demonstrations are more likely to be feasible. Specifically, we create probability distribution pj over demonstrators:
37 Eese w(())
Σド=
1 Ei Egiess w(E))
We repeatedly and independently query the demonstrator j according Py and collect demonstration. The proposed sampling strategy samples more demonstrations from closer demonstrators. We empirically show that the sampling strategy derived from our feasibility performs better than uniform sampling or using other feasibility metrics as in [7].
Experimental Results
We experiment with four MuJoCo environments, a simulated Franka Panda Arm, and a real Franka Panda Arm. We also show results various compositions of demonstrations of different dynamics and the performance gain when we are given larger budget to collect demonstrations. We compare our approach with a standard imitation learning algorithm: GAIL [4], imitation learning from demonstrations with different dynamics methods without a measure of feasibility: SAIL [5], and with a feasibility score: ID-FFeas [7], which uses an inverse dynamics model estimate feasibility.
5.1 MuJoCo Experiments
The swimmer agent has three links and two joints. The goal of the agent is to move forward by rotating the joints. As shown in Fig. 3(a), we create different dynamics by setting the joint limit of the front and the back joints, denoted by (a,, og). The original Swimmer environment
Swimmer.
has (ar, (t)) = (100", 100%). We create four demonstrator environments (a,, (n)): (i) (100", 12%), (ii) (100e, 20"), (iii) (100", 100%). and (iv) (10", 100%). We also create the imitator environment by setting (ay, = (100%, 109). The demonstrators (i) and (ii) are closer to the imitator terms of their dynamics, while the demonstrators (iii) and (iv) are farther.

---

1,00
5000
0.75
Stccess
0,50
0.25
5500
(a) Dynamies
(b) Expected Return
Success Rate
(d) Sample Trajectories
Ours
ID-Feas
SAIL
GAIL
(a) Illustration different dynamics in the real robot arm environment. (b-e) The bar plots show the expected return and success rate compared to other methods. (d) Sampled trajectories using different methods.
Figure 4:
Walker2d. The Walker2d is an agent with two legs where each leg consists of 3 joints. We create different dynamics by using different frictions 3 for the feet, i.e., the link that touches the ground. The original Walker2d uses 3 = 0.9. We create two settings to show high friction and low friction of the imitator with mix of frictions for the demonstrators. In the first setting, there are four demonstrators: (i) B = 19.9, (ii) B = 9.9, (iii) 33 0.9, and (iv) 3 = 0.7. The imitator has 3 24.9. In the second setting, there are four demonstrators: (i) 29.9, (ii) B = 19.9, (iii) 3 = 1.1, and (iv) 3 0.7. The imitator has A = 0.9.
The HalfCheetah is an agent with two legs the front and back of the body, where each leg consists of three joints. We create different dynamics by varying the control force limit of joints of the front leg and back leg, where we multiply factor Ty with the original control force limit of the front leg and multiply %/ with the limit of the back leg. We create two settings, where the demonstrators have low and high similarity with each other. In the first setting, there are four demonstrators with (7f: (h)): (i) (0.05, 1), (ii) (0.5, 1). (iii) (1, 0.5), and (iv) (1, 0.05). The imitator has (7/: 76) = (0.01, 1). In the second setting, there are four demonstrators with (T5, () (0.01,1) 1). (ii) (0.05, 1), (iii) (1, 0.05), and (iv) (1, 0.01.. The imitator has (7y, 76) 10.
HallCheetah.
Hopper. The Hopper is an agent with one leg consisting of 3 joints. We create different dynamics by using different gravitational constants g. The original Hopper uses g 9.81. We create four demonstrator environments: (i) g 20.0, (ii) g 9.81, (iii) g 5.0, and (iv) g 2.0. We also create the imitator environment by setting g = 15.0.
The detailed composition of demonstrations for all four environments is included in the Appendix. For all the Mujoco environments, we evaluate the expected return each policy by rolling out 100 trajectories in the environment with the policy and compute the average expected return of the 100 trajectories. We run each experiment for 5 times and show the mean and the standard deviation.
Results. We show the expected return with respect to the number of steps for the four different envi- ronments in Fig. 3. We show the results of the second setting for the Walker2d and the HalfCheetah in the Appendix. We observe that our proposed feasibility achieves the best performance among all the methods. The highest p-value comparing our method to baselines is 0.116 with ID-Feas for Swimmer, 2.55e 14 with GAIL for Walker2d, 0.188 with ID-Feas for HalfCheetah, and 0.026 with GAIL for Hopper. In particular, our method outperforms ID-Feas, which indicates that the proposed feasibility more accurately filters out far from feasible demonstrations. SAIL performs even worse than GAII,, this is because SAIL. can more strictly follow the state sequences of demonstrations than GAIL including those far from feasible demonstrations, Our demonstration set is composed of high percentage of demonstrations from unrelated dynamics, which can mislead SAIL's learned policy.
5.2 Simulated and Real Robot Arm Experiments
Setup. We create a simulated robot arm based on a Panda Robot Arm implemented in the PyBul- let [30] and a real robot arm environment using a Franka Panda Arm We include the results for the simulated robot arm in the Appendix. As shown in Fig. 4(a), create task of moving book to the shelf but the closest region on the shelf is full. So we need to move the book to the empty area of the shelf without colliding with the shelf and the existing books on the shelf. We create two dynamics for the robot arm: using a 7-DoF control which can move freely in the 3D space, and using
httpps//www.rranka..
7

---

1.00
Expeçted Iteturs
30n0
Exprected Detuum
Baso Apppection Returrs
Expected
50
2000
2000
2000
1000
200
100
200
200 Number of Demonstrations
400
400 Number Demonstrations (d) Hopper
200
Number of Demonstrations (a) Swimmer
Number Demonstrations
(b) Walker2d
(c) HalfCheetah
Ours
ID-Feess
SAIL
GAIL.
Figure 5: (a-d) The expected return when increasing the number of demonstrations from agents with unrelated dynamies. The results in Fig. correspond using 500 demonstrations from each unrelated dynamics. In both of these settings, there will also be fixed number of demonstrations from agents with related dynamics shown in Appendix.
a 3-DoF control, which limited to moving on the red plane area. We collect demonstrations from both 7-DoF and 3-DoF controllers and aim to learn optimal policy for the 3-DoF robot.
For evaluation, we use two metries: (1) The expected return based on a reward penalizing collision with the shelf and existing books while rewarding the successfully moving the book to the empty area of the shelf within the time limit. More detail on the reward the Appendix. (2) The success rate of finishing the task over 100 trials.
We observe that the proposed approach outperforms the baseline methods both expected return and success rate as shown in Fig. 4. The highest p-value for the expected return is 2.432 x 10-? and for the success rate is 3.534 10-8 (both with ID-Feas), which are statistically significant.
5.3 Analysis
We conduct experiments with varying compositions of demonstrations and investigate the performance of different approaches when have the budget to acquire additional demonstrations, We show the results of varying the number of demonstrations from all demonstrators in the Appendix.
Varying the Number of Demonstrations from each Unrelated Demonstrator, For the first three experiment settings in the Mujoco environment, we have two demonstrators with similar dynamics to the imitator and two demonstrators with far apart dynamies. We vary the number of demonstrations from the far apart demonstrators to investigate their influence on the different methods. We conduct experiments the first setting for the Swimmer, Walker2I), HalfCheetah, and Hooper and report the results in Fig. (a)), (b)), 5(c) and 5(d). With an increasing number of demonstrations from the far apart demonstrators, the expected return of all the methods decreases, while our method shows the best performance consistently across different numbers of demonstrations. This demonstrates that our feasibility can effectively filter far from feasible demonstrations and ensure the policy learns from useful demonstrations.
Conclusion
Summary. We propose an algorithm to learn feasibility metric imitate demonstrations drawn from agents with different dynamics. Our feasibility metric captures how likely it is for each demonstration to feasible the imitator. We develop feasibility MDP (f-MDP) and derive feasibility by learning the optimal policy for the f-MDP. We show that the policy learned from the demonstrations reweighted by the proposed fcasibility score outperforms other imitation learning methods in various environments and different compositions of demonstrations,
Limitations and Future Work. Our work only addresses the problem of filtering out far from feasible demonstrations, but does not solve the problem of learning a policy from those feasible but suboptimal or nearly feasible demonstrations from different dynamics. There are situations where demonstrations are feasible but not optimal for the imitator, especially when the ability of the demonstrator more restricted than the imitator. In the future, we aim to study these more general settings.

---

Acknowledgments
We would like to thank FLI grant RFP2-000, NSF Awards 1941722 and 1849952, and DARPA HiCoN project for their support of this project.
References
[1] Abbeel and Y. Ng. Apprenticeship learning via inverse reinforcement learning. In ICML, pages 1-8, 2004.
[2] B. D. Ziebart, A. I.. Maas, A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforcement learning. In AAAI, 2008.
[3] A. H. Qureshi, B. Boots, and M. C. Yip. Adversarial imitation via variational inverse reinforce- ment learning. In ICLR, 2019.
[4] J. Ho and S. Ermon. Generative adversarial imitation learning. In NeurlPS, volume 29, 2016.
[5] F. Liu, Z. Ling, T. Mu, and H. Su. State alignment-based imitation learning. In ICLR, 2019.
[6] Radosavovic, X. Wang, L. Pinto, and Malik. State-only imitation learning for dexterous manipulation. arXiv preprint arXiv:2004.046550 2020.
[7] Z. Cao and D. Sadigh. Learning from imperfect demonstrations from agents with varying dynamics. IEEE Robotics and Automation Letters (RA-L), 2021.
[8] M. Bain and C. Sammut. framework for behavioural cloning. In Machine Intelligence 15, 1995.
[9] S. Ross, G. Gordon, and D. Bagnell. reduction of imitation learning and structured prediction to no-regret online learning. In AISTATS, 2011.
[10] H. Daumé, J. Langford, and D. Marcu, Search-based structured prediction. Machine learning, 2009.
[11] Ross and D. Bagnell. Efficient reductions for imitation learning. In A/STATS, 2010.
[12] A. Y. Ng. S. J. Russell, et al. Algorithms for inverse reinforcement learning. In ICML,, pages 663-670, 2000.
[13] Finn, S. Levine, and P. Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International conference on machine learning, pages 49-58. PMLR, 2016.
[14] F. Torabi, G. Warnell, and P. Stone. Behavioral cloning from observation. In IJCAI, pages 4950-4957, 2018.
[15] Y. Schroecker and C. L. Isbell. State aware imitation learning. In NeuriPS, 2017.
[16] Torabi, Warnell, and Stone. Generative adversarial imitation from observation. Imitation, Intent, and Interaction (13) Workshop at ICML., 2019.
[17] W. Sun, A. Vemula, B. Boots, and D. Bagnell. Provably efficient imitation learning from observation alone. ICML, 2019.
[18] C. L. Nehaniv, K. Dautenhahn, et al. The correspondence problem. Imitation in animals and artifacts, 41, 2002.
[19] B. D. Argall, S. Chernova, M. Veloso, and B. Browning. survey of robot learning from demonstration. Robotics and autonomous systems, 2009.
[20] P. Englert, A. Paraschos, J. Peters, and M. P. Deisenroth. Addressing the correspondence problem by model-based imitation learning. In ICRA Workshop on Autonomous Learning. 2013.
[21] Calinon, F. Guenter, and A. Billard. On learning, representing, and generalizing a task in humanoid robot. TSMC, 2007.

---

[22] C. Eppner, J. Sturm, M. Bennewitz, C. Stachniss, and W. Burgard. Imitation learning with generalized task descriptions. In ICRA, 2009.
[23] Tobin, R. Fong, A. Ray, Schneider, W. Zaremba, and Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RS/ international conference on intelligent robots and systems (IROS), pages 23-30. IEEE, 2017.
[24] X. B. Peng, M. Andrychowicz, W. Zaremba, and Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference robotics and automation (ICRA), pages 3803-3810, IEEE, 2018.
[25] S. Zakharov, W. Kehl, and S. Ilic. Deceptionnet: Network-driven domain randomization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 532-541, 2019.
[26] Q. Zhang. T. Xiao, A. A. Efros, I.. Pinto, and X. Wang. Learning cross-domain correspon- dence for control with dynamics cycle-consistency. International Conference on Learning Representations, 2021. URL httpps:/opeereviiee
[27] P. Sharma, L. Mohan, L. Pinto, and A. Gupta. Multiple interactions made easy (mime): Large scale demonstrations data for imitation. In Conference on robot learning, pages 906-915. PMLR, 2018.
[28] T. Gangwani and J. Peng. State-only imitation with transition dynamics mismatch. In Inter- national Conference on Learning Representations, 2020. URL htt.httts///peererrii net/ DorumPid=KUgLLyyrYee
[29] Y.-H. Wu, N. Charoenphakdee, H. Bao, Tangkaratt, and M. Sugiyama, Imitation learning from imperfect demonstration. In ICML, 2019.
[30] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. 2016.
10


Reviews:
[{"review_id": "jTPqpaCVuhX", "reviewer": "Reviewer_fG77", "summary": "This paper attempts to improve the algorithm from the perspective of Feasibility of Imitate Learning, and encourage the imitator to learn from more informative demonstrations by defining a new reward for the learning task of Feasibility. In addition, the theory in the article is preliminarily verified through simulation experiments and real robotic arm experiments. The results are basically credible and have some positive effects on the development of this field.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper attempts to introduce the concept of feasibility Markov Decision Process (f-MDP), and  to find a solution to the problem of Imitate Learning with Different Dynamics from a higher perspective. This is beneficial for obtaining new ideas. However, in the process of exploring feasibility score or metric, the use of norm and parameter gamma is extremely similar to the traditional RL method. What’s more, this paper uses distance as a feasibility evaluation metric, which may in fact do not sufficiently reflect feasibility, even it seems more like following the demonstration trajectory. Actually, the f-MDP is very similar to ordinary MDP, and there is little obvious difference except for the description.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "It is recommended to adjust the idea and experimental plan to make the concept of feasibility more prominent and reasonable.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "1. It is not reasonable enough that this paper uses robotic arms with different degrees of freedom to cite its ideas. In robotics, if the robot arm can be manipulated to complete the RL task, the dynamic model of the robot arm is almost known, at least it is not so difficult to obtain the dynamic model.\n2.In Figure 1, it should use the robot arm picture consistent with the experiment, instead of the red robot arm in the picture (it can use the Mark-On-Image method to emphasize 3-DOF or 7-DOF).\n3.Line 134-135, the statement about perfect inverse dynamics model is not clear enough.\n4.Line 179，if the Eqn.2 is not used here, the new formula should be mentioned in the text as soon as possible. For readers, the final method used in this idea is very important.\n5.Line 185, for the traditional RL reward function, the use of gamma is the basis of RL theory, so the statements in the article should be reduced as much as possible. \n"}}, {"review_id": "iq5MC1scv7N", "reviewer": "Reviewer_GBdL", "summary": "The paper is concerned with learning movement primitives from demonstrations collected on robots with dynamics that are different than that of the imitator robot (similar to the correspondence problem). Specifically, the paper contributes a method to compute a metric that quantifies the feasibility of the imitator robot reproducing a given demonstration. The available demonstrations are then weighted based on this feasibility metric so that the imitator can prioritize demonstrations are more suitable for the imitator's dynamics. Experimental results (both in simulation a real robot platform) demonstrate the utility of the proposed approach and its and relative benefits over existing approaches.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n+ The work is well motivated by the need to leverage already available demonstrations, especially when they are collected on a robot with a different morphology/dynamics.\n+ The paper is well written and was a joy to read.\n+ The contributions are sufficiently contextualized within the existing literature on imitation learning.\n+ The method does not assume knowledge of the exact inverse dynamics model of the imitator, and relies only on knowledge of the imitator's environment.\n+ By moving away from a binary feasibility function, the proposed method does not completely discard infeasible trajectories that could still contain useful information.\n+ The figures in the paper are well designed and composed. They prove to be very helpful illustrations of various core concepts in the paper.\n+ The probability distribution computed in Eq. (4) that models the usefulness of a demonstrators is very helpful from a practical stand point as we can focus our energy on collecting additional demonstrations from more useful platforms if necessary.\n+ The experiments are well designed and conclusively demonstrate the utility of the proposed approach. Further, careful comparison with SOTA approaches illustrate the relative benefits that the proposed approach brings to the table. Further, the analysis in Section 5.3 is illuminating and commendable. \n\n**Weaknesses**\n\n- As the paper acknowledges, the idea of learning a feasibility score is not novel and has been proposed in Ref. [7]. Having said that, this is a minor weakness for a couple of reasons. First, it is important to note that Ref. [7] was published only in 2021. Second, the paper explicitly includes Ref. [7] as a baseline in the experiment and demonstrates the superiority of the proposed method.\n- A key assumption made by this work is that \"by imitating useful demonstrations, we can learn the optimal policy\". While this assumption is a direct result of the no-free-lunch theorem given the problem setup, it is currently buried at the end of Section 3. This assumption should be acknowledged more prominently and its consequences explicitly discussed.\n- While the experiments compare the performance of the proposed approach against some existing imitation learning algorithms, it does not compare against approaches that map demonstrations to the imitator. Granted that the demonstrations might not be feasible and the mapping might produce demonstrations are far from the intended demonstrations. As the paper claims, such mapping \"may lead to unknown behavior\".  But, it would be very helpfully experimentally verify this claim and demonstrate that the proposed method (feasibility-based weighting) indeed performs better than a policy learned from mapped (and potentially problematic) demonstrations.\n- The methods assumes that a single number uniformly defines the feasibility of a demonstrated trajectory. What if parts of the demonstrations are useful/feasible, while other parts are harmful/infeasible?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper presents a much-needed approach to handling demonstrations that are collected from a heterogeneous set of robot platforms that have potentially different dynamics. Importantly, the approach leverages information from such demonstrations without needed the exact inverse dynamics model. Save for some minor weaknesses, this is a solid conference paper that presents an approach that will likely have a broad impact. While the paper is mostly focused on learning primitives for manipulators. The key ideas are much broader and can be extended to various applications, such as autonomous vehicles and multi-agent systems.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "1. How do we know that the assumption \"by imitating useful demonstrations, we can learn the optimal policy\" is valid for a given setting? How strong is it? What are the consequences of the assumption being false?\n\n2. In the introduction, please provide a brief and conceptual explanation of what it means to assume access to the \"environment\" of the imitator. For instance, state how the environment is modeled, and what is assumed to be given.\n\n3. What if the assumption that the reproductions that initially diverge are going to diverge more in the later time step is wrong. For instance, what if the \"closest\" or \"best\" reproduction that the imitator could produce requires the imitator to deviate from the demonstration in the very initial time steps of the demonstrations and then converge back close to the demonstrations? In other words, how valid is the assumption that a discount factor is needed?\n\n4. On a related note, what if parts of the demonstrations are useful/feasible, while other parts are harmful/infeasible? Could the score be adapted to address this?\n\n5. The narrative in the paper makes it seem like Ref. [7] just assumes an accurate inverse dynamics model. But, from a quick glance, it appears that Ref. [7] in fact learns the inverse dynamics model. If this is indeed the case, please clarify in the manuscript it avoid unintentionally misleading the readers."}}, {"review_id": "7jFLSA0L-vX", "reviewer": "Reviewer_Srqj", "summary": "The paper considers the problem of imitation learning with demonstrations from potentially different dynamics. The authors propose a new feasibility score which evaluates how feasible it is to reproduce the demonstration with the imitator's dynamics. The feasibility score is obtained from an f-MDP, which shares the state space, action space, transition function with the imitator, but has a reward function quantifying state deviation from the demonstrations. This feasibility can be further used in a more interactive setting to determine which demonstrator to query new trajectories. The authors show in various tasks that the newly proposed feasibility score can help the learner rule out infeasible demonstrations and hence achieve better task performance. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper considers an important problem of learning with demonstrations from potentially different dynamics. The problem itself is well-motivated: The introduction of the paper provides a concise but convincing example. \n- The authors provide a comprehensive survey over related work, both broadly in the field of imitation learning, as well as the specific area of learning with demonstrations from different dynamics. The authors also pinpoint the difference between their new approach and existing work. \n- The experimental evaluation is thorough. The author compare their proposed algorithm with state-of-the-art methods in the area and provide convincing results that the proposed method generally outperforms baselines in these tasks. \n\nWeaknesses:\n- The paper is overall technically sound. I like the f-MDP formation a lot since I think it is more suitable for stochastic MDPs than learning an inverse dynamics function, which is commonly used in the literature. However, there seems to be occasions where the authors use treatments that are not entirely appropriate for stochastic MDPs. For example, in line 113, the expectation should be taken w.r.t. the transition probability as well; in line 141, enforcing s_{t+1}^{I} = s_{t+1}^d might not be feasible for stochastic system, as it is sampled from a distribution; in line 187, \"feasible trajectories still receive the maximal reward of 0\" might not be true for stochastic MDPs based on the definition of feasibility from the authors. The authors are encouraged to go over these statements and potentially modify them in the next version of the manuscript. \n- A potential weakness is that solving the f-MDP might require quite some environment interactions and computation. But it seems like the proposed method still outperforms existing methods given the same amount of interactions. But it would be great if the authors can provide more explanations on this. For example, how are the number of interactions for solving f-MDP and that for learning an inverse dynamics model compared to each other?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper proposed to solve an important and practical problem of learning with demonstrations from different dynamics. The paper is well-written and overall technically sound (with a few exceptions mentioned in the comments). The literature review is thorough and the novelty is clear. The experimental results are convincing. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "- The authors are advised to modify a few statements so that they are true for stochastic MDPs, as is covered in the \"Weaknesses\" part.\n- In equation (1), the formulation seems to be equivalent to taking the expectation with respect to the stochastic transition. This may potentially have some undesired consequences. Consider an example where we have that with probability (1-\\epsilon), the next state s_{t+1}^i matches with the demonstration exactly, but with probability \\epsilon, the next state deviates a lot from the demonstration, e.g. ||s_{t+1}^i - s_{t+1}^d||_2 -> infinity. According to equation (1), this demonstration would be really bad because, in expectation, the deviation is high (due to the large deviation with extremely small probability). Yet given the definition of feasibility defined in the paper, this demonstration is perfectly feasible (there exist actions which recover the demonstration with probability > 0, in fact with high probability). It would be great if the author can motivate this design choice of using the L-2 norm and potentially provide a few more choices of the reward function. \n- It would be great if the authors can elaborate how the discount factor \\gamma^f is chosen and what are the practical considerations regarding it. \n- Does T contain all demonstrations or only a single demonstration? \n- If there are two trajectories that happen to have the same state transition pair (s_t^d, s_{t+1}^d), but one is feasible and the other is not. Then what would w((s_t^d, s_{t+1}^d)) be in this case?\n- What is the expected returns for the demonstrations in all the experiments? Are they higher than that of the proposed method or they are comparable?\n- How is success rate defined for the Franka robot experiment?\nMinor issues:\n- Line 124: The authors mentioned that [7] requires \"access to accurate inverse dynamics model\". Although it seems like [7] proposes to learn the inverse dynamics model. The authors might want to clarify that in the later version. \n- Line 156: The curly bracket should be closed after \"\\inT\".\n- In the appendices: what is a \"disabled\" panda robot?\n"}}]

Author rebuttal:

Meta-review: This paper presents an approach for learning from demonstrations that allows the imitator to distinguish demonstrations feasible for their specific action spaces and the slight difference between the dynamics of demonstrators and imitators. Some strong statements are not supported by the experimental results, e.g. "we would like to be able to leverage demonstrations collected from agents with different dynamics." In the HalfCheetah, Swimmer examples the actions space of the imitator and demonstrator are different whereas in the Walker2d the friction coefficient between the demonstration and mutation scenarios are different. 
The idea of identifying the demonstrations for a specific kinematic and dynamic chain of imitator is very relevant CoRL scope. Reviewers raised some concerns regarding the presentation of the paper and the corresponding concepts. For instance, there are similarities between the proposed approach and traditional RL. Moreover, the L2-norm distance between the demonstration and imitation trajectories may not be always useful in robotic tasks: (1) the trajectories may not be aligned temporally — which is not discussed in this paper; and (2) the L2 norm distance may not sufficiently reflect the feasibility (a) trajectories either are expressed in task-space or (b) the trajectories are expressed in join space, hence the L-2 norm distances for orientation and position or the distance between 3 DOF and 7 DOF must be clarified. A clear description of the problem definition and problem formulation can improve the quality of the paper.

The limitation of the approach must be clearly discussed. It should be clarified to what extent the approach can deal with different dynamics: (1) is it only reduced DOF of a manipulator the change in the dynamic, (2) or the change of friction coefficient in the interactive tasks; or (3) the change of dynamic model, e.g. the Mass, Coriolis and Gravity Matrices are different. 


**Quality**: Needs improvement. Some claims are not fully supported by the results which can be improved. E.g. while the demonstrator and imitator manipulators in the motivating example of Fig. 1 have different kinematic chains, the experimental results only show a Franka arm as the demonstrator (with 3 and 7 DOFs) and imitator (with 3 DOF) where their kinematic chain is the same. Some additional experiments or analyses are needed to support the claims about the use of the approach for different dynamics— this term needs a clear definition at the beginning of the paper. 

**Clarity**: The paper is clear after addressing the concerns raised by the reviewers.  


**Significance**: The paper has practical significance and the theoretical significance is clear after response to reviewers. 

**Originality**: The author presents a novel idea which computes the feasibility metric for a kinematic/dynamic chain and a demonstration. Nonetheless, the authors need to describe what is the similarity and differences between the proposed approach and traditional RL.  


Decision: Accept (Poster)