Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper is concerned with learning movement primitives from demonstrations collected on robots with dynamics that are different than that of the imitator robot (similar to the correspondence problem). Specifically, the paper contributes a method to compute a metric that quantifies the feasibility of the imitator robot reproducing a given demonstration. The available demonstrations are then weighted based on this feasibility metric so that the imitator can prioritize demonstrations are more suitable for the imitator's dynamics. Experimental results (both in simulation a real robot platform) demonstrate the utility of the proposed approach and its and relative benefits over existing approaches.
strengths_and_weaknesses: **Strengths**

+ The work is well motivated by the need to leverage already available demonstrations, especially when they are collected on a robot with a different morphology/dynamics.
+ The paper is well written and was a joy to read.
+ The contributions are sufficiently contextualized within the existing literature on imitation learning.
+ The method does not assume knowledge of the exact inverse dynamics model of the imitator, and relies only on knowledge of the imitator's environment.
+ By moving away from a binary feasibility function, the proposed method does not completely discard infeasible trajectories that could still contain useful information.
+ The figures in the paper are well designed and composed. They prove to be very helpful illustrations of various core concepts in the paper.
+ The probability distribution computed in Eq. (4) that models the usefulness of a demonstrators is very helpful from a practical stand point as we can focus our energy on collecting additional demonstrations from more useful platforms if necessary.
+ The experiments are well designed and conclusively demonstrate the utility of the proposed approach. Further, careful comparison with SOTA approaches illustrate the relative benefits that the proposed approach brings to the table. Further, the analysis in Section 5.3 is illuminating and commendable. 

**Weaknesses**

- As the paper acknowledges, the idea of learning a feasibility score is not novel and has been proposed in Ref. [7]. Having said that, this is a minor weakness for a couple of reasons. First, it is important to note that Ref. [7] was published only in 2021. Second, the paper explicitly includes Ref. [7] as a baseline in the experiment and demonstrates the superiority of the proposed method.
- A key assumption made by this work is that "by imitating useful demonstrations, we can learn the optimal policy". While this assumption is a direct result of the no-free-lunch theorem given the problem setup, it is currently buried at the end of Section 3. This assumption should be acknowledged more prominently and its consequences explicitly discussed.
- While the experiments compare the performance of the proposed approach against some existing imitation learning algorithms, it does not compare against approaches that map demonstrations to the imitator. Granted that the demonstrations might not be feasible and the mapping might produce demonstrations are far from the intended demonstrations. As the paper claims, such mapping "may lead to unknown behavior".  But, it would be very helpfully experimentally verify this claim and demonstrate that the proposed method (feasibility-based weighting) indeed performs better than a policy learned from mapped (and potentially problematic) demonstrations.
- The methods assumes that a single number uniformly defines the feasibility of a demonstrated trajectory. What if parts of the demonstrations are useful/feasible, while other parts are harmful/infeasible?
summary_of_recommendation: This paper presents a much-needed approach to handling demonstrations that are collected from a heterogeneous set of robot platforms that have potentially different dynamics. Importantly, the approach leverages information from such demonstrations without needed the exact inverse dynamics model. Save for some minor weaknesses, this is a solid conference paper that presents an approach that will likely have a broad impact. While the paper is mostly focused on learning primitives for manipulators. The key ideas are much broader and can be extended to various applications, such as autonomous vehicles and multi-agent systems.
issues: 1. How do we know that the assumption "by imitating useful demonstrations, we can learn the optimal policy" is valid for a given setting? How strong is it? What are the consequences of the assumption being false?

2. In the introduction, please provide a brief and conceptual explanation of what it means to assume access to the "environment" of the imitator. For instance, state how the environment is modeled, and what is assumed to be given.

3. What if the assumption that the reproductions that initially diverge are going to diverge more in the later time step is wrong. For instance, what if the "closest" or "best" reproduction that the imitator could produce requires the imitator to deviate from the demonstration in the very initial time steps of the demonstrations and then converge back close to the demonstrations? In other words, how valid is the assumption that a discount factor is needed?

4. On a related note, what if parts of the demonstrations are useful/feasible, while other parts are harmful/infeasible? Could the score be adapted to address this?

5. The narrative in the paper makes it seem like Ref. [7] just assumes an accurate inverse dynamics model. But, from a quick glance, it appears that Ref. [7] in fact learns the inverse dynamics model. If this is indeed the case, please clarify in the manuscript it avoid unintentionally misleading the readers.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper attempts to improve the algorithm from the perspective of Feasibility of Imitate Learning, and encourage the imitator to learn from more informative demonstrations by defining a new reward for the learning task of Feasibility. In addition, the theory in the article is preliminarily verified through simulation experiments and real robotic arm experiments. The results are basically credible and have some positive effects on the development of this field.
strengths_and_weaknesses: This paper attempts to introduce the concept of feasibility Markov Decision Process (f-MDP), and  to find a solution to the problem of Imitate Learning with Different Dynamics from a higher perspective. This is beneficial for obtaining new ideas. However, in the process of exploring feasibility score or metric, the use of norm and parameter gamma is extremely similar to the traditional RL method. What’s more, this paper uses distance as a feasibility evaluation metric, which may in fact do not sufficiently reflect feasibility, even it seems more like following the demonstration trajectory. Actually, the f-MDP is very similar to ordinary MDP, and there is little obvious difference except for the description.
summary_of_recommendation: It is recommended to adjust the idea and experimental plan to make the concept of feasibility more prominent and reasonable.
issues: 1. It is not reasonable enough that this paper uses robotic arms with different degrees of freedom to cite its ideas. In robotics, if the robot arm can be manipulated to complete the RL task, the dynamic model of the robot arm is almost known, at least it is not so difficult to obtain the dynamic model.
2.In Figure 1, it should use the robot arm picture consistent with the experiment, instead of the red robot arm in the picture (it can use the Mark-On-Image method to emphasize 3-DOF or 7-DOF).
3.Line 134-135, the statement about perfect inverse dynamics model is not clear enough.
4.Line 179，if the Eqn.2 is not used here, the new formula should be mentioned in the text as soon as possible. For readers, the final method used in this idea is very important.
5.Line 185, for the traditional RL reward function, the use of gamma is the basis of RL theory, so the statements in the article should be reduced as much as possible.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper considers the problem of imitation learning with demonstrations from potentially different dynamics. The authors propose a new feasibility score which evaluates how feasible it is to reproduce the demonstration with the imitator's dynamics. The feasibility score is obtained from an f-MDP, which shares the state space, action space, transition function with the imitator, but has a reward function quantifying state deviation from the demonstrations. This feasibility can be further used in a more interactive setting to determine which demonstrator to query new trajectories. The authors show in various tasks that the newly proposed feasibility score can help the learner rule out infeasible demonstrations and hence achieve better task performance. 
strengths_and_weaknesses: Strengths:
- The paper considers an important problem of learning with demonstrations from potentially different dynamics. The problem itself is well-motivated: The introduction of the paper provides a concise but convincing example. 
- The authors provide a comprehensive survey over related work, both broadly in the field of imitation learning, as well as the specific area of learning with demonstrations from different dynamics. The authors also pinpoint the difference between their new approach and existing work. 
- The experimental evaluation is thorough. The author compare their proposed algorithm with state-of-the-art methods in the area and provide convincing results that the proposed method generally outperforms baselines in these tasks. 

Weaknesses:
- The paper is overall technically sound. I like the f-MDP formation a lot since I think it is more suitable for stochastic MDPs than learning an inverse dynamics function, which is commonly used in the literature. However, there seems to be occasions where the authors use treatments that are not entirely appropriate for stochastic MDPs. For example, in line 113, the expectation should be taken w.r.t. the transition probability as well; in line 141, enforcing s_{t+1}^{I} = s_{t+1}^d might not be feasible for stochastic system, as it is sampled from a distribution; in line 187, "feasible trajectories still receive the maximal reward of 0" might not be true for stochastic MDPs based on the definition of feasibility from the authors. The authors are encouraged to go over these statements and potentially modify them in the next version of the manuscript. 
- A potential weakness is that solving the f-MDP might require quite some environment interactions and computation. But it seems like the proposed method still outperforms existing methods given the same amount of interactions. But it would be great if the authors can provide more explanations on this. For example, how are the number of interactions for solving f-MDP and that for learning an inverse dynamics model compared to each other?
summary_of_recommendation: The paper proposed to solve an important and practical problem of learning with demonstrations from different dynamics. The paper is well-written and overall technically sound (with a few exceptions mentioned in the comments). The literature review is thorough and the novelty is clear. The experimental results are convincing. 
issues: - The authors are advised to modify a few statements so that they are true for stochastic MDPs, as is covered in the "Weaknesses" part.
- In equation (1), the formulation seems to be equivalent to taking the expectation with respect to the stochastic transition. This may potentially have some undesired consequences. Consider an example where we have that with probability (1-\epsilon), the next state s_{t+1}^i matches with the demonstration exactly, but with probability \epsilon, the next state deviates a lot from the demonstration, e.g. ||s_{t+1}^i - s_{t+1}^d||_2 -> infinity. According to equation (1), this demonstration would be really bad because, in expectation, the deviation is high (due to the large deviation with extremely small probability). Yet given the definition of feasibility defined in the paper, this demonstration is perfectly feasible (there exist actions which recover the demonstration with probability > 0, in fact with high probability). It would be great if the author can motivate this design choice of using the L-2 norm and potentially provide a few more choices of the reward function. 
- It would be great if the authors can elaborate how the discount factor \gamma^f is chosen and what are the practical considerations regarding it. 
- Does T contain all demonstrations or only a single demonstration? 
- If there are two trajectories that happen to have the same state transition pair (s_t^d, s_{t+1}^d), but one is feasible and the other is not. Then what would w((s_t^d, s_{t+1}^d)) be in this case?
- What is the expected returns for the demonstrations in all the experiments? Are they higher than that of the proposed method or they are comparable?
- How is success rate defined for the Franka robot experiment?
Minor issues:
- Line 124: The authors mentioned that [7] requires "access to accurate inverse dynamics model". Although it seems like [7] proposes to learn the inverse dynamics model. The authors might want to clarify that in the later version. 
- Line 156: The curly bracket should be closed after "\inT".
- In the appendices: what is a "disabled" panda robot?
review_title: null

Meta-review: This paper presents an approach for learning from demonstrations that allows the imitator to distinguish demonstrations feasible for their specific action spaces and the slight difference between the dynamics of demonstrators and imitators. Some strong statements are not supported by the experimental results, e.g. "we would like to be able to leverage demonstrations collected from agents with different dynamics." In the HalfCheetah, Swimmer examples the actions space of the imitator and demonstrator are different whereas in the Walker2d the friction coefficient between the demonstration and mutation scenarios are different. 
The idea of identifying the demonstrations for a specific kinematic and dynamic chain of imitator is very relevant CoRL scope. Reviewers raised some concerns regarding the presentation of the paper and the corresponding concepts. For instance, there are similarities between the proposed approach and traditional RL. Moreover, the L2-norm distance between the demonstration and imitation trajectories may not be always useful in robotic tasks: (1) the trajectories may not be aligned temporally — which is not discussed in this paper; and (2) the L2 norm distance may not sufficiently reflect the feasibility (a) trajectories either are expressed in task-space or (b) the trajectories are expressed in join space, hence the L-2 norm distances for orientation and position or the distance between 3 DOF and 7 DOF must be clarified. A clear description of the problem definition and problem formulation can improve the quality of the paper.

The limitation of the approach must be clearly discussed. It should be clarified to what extent the approach can deal with different dynamics: (1) is it only reduced DOF of a manipulator the change in the dynamic, (2) or the change of friction coefficient in the interactive tasks; or (3) the change of dynamic model, e.g. the Mass, Coriolis and Gravity Matrices are different. 


**Quality**: Needs improvement. Some claims are not fully supported by the results which can be improved. E.g. while the demonstrator and imitator manipulators in the motivating example of Fig. 1 have different kinematic chains, the experimental results only show a Franka arm as the demonstrator (with 3 and 7 DOFs) and imitator (with 3 DOF) where their kinematic chain is the same. Some additional experiments or analyses are needed to support the claims about the use of the approach for different dynamics— this term needs a clear definition at the beginning of the paper. 

**Clarity**: The paper is clear after addressing the concerns raised by the reviewers.  


**Significance**: The paper has practical significance and the theoretical significance is clear after response to reviewers. 

**Originality**: The author presents a novel idea which computes the feasibility metric for a kinematic/dynamic chain and a demonstration. Nonetheless, the authors need to describe what is the similarity and differences between the proposed approach and traditional RL.  


Decision: Accept (Poster)