Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: Authors propose a novel learning from demonstration (LfD) framework MAIL which allows for teacher and student morphologies to vary greatly. The particular discrepency they focus on are when the number of end effectors differ. Their method consists of learning a dynamics model which is used by a trajectory optimiser to convert demonstrations into the student's morphology. Finally, this trajectory is used by an existing LfD method which learns a policy in state and image space, intended to generalise to arbitrary initial environment states. They demonstrate their method in a wide range of experimental scenarios, both in simulation and hardware, and show their method outperforms baseline methods and is able to generalise to different morphologies.
review: Quality/ Clarity
--------------------
Paper is overall well written and presented. However, some sections are too reliant on appendices and cited work. For example, a short description of DMfD would help to make the paper more self-contained. Furthermore, the abalation study is not really useful without the results. Either incorporate results into the section or move entirely into appendix.

Originality
---------------
The proposed method is mainly a combination of existing methods. Utilising full state information for task transfer and then training the policy on images is an interesting idea but may be restrictive for certain problem settings.

Significance
-----------------
The proposed framework is shown to work well and is mostly practical enough to be utilised. Being able to demonstrate a task naturally and transfer to a robot with a significantly different embodyment has real use-case scenarios.

Strengths
----------------
- Method works well and generalises to varying inital conditions
- Thorough experiments and shown to transfer to real world hardware
- Video is nicely put together and helps in understanding the paper

Weaknesses
-------------------
- Question around novelity since this is mainly a trivial combination of existing work
- Applicability to certain problem settings may be limited due to requiring full state information
- Method explanations are lacking and could be more self contained rather than simply citing work being used
- Some sections rely too much on content in appendix


robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The main strength of this paper is the experimental section. However, overall the paper lacks originality combined with vague method descriptions.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper presents MAIL, a new imitation learning algorithm for learning across large differences in teacher and student action spaces. For example, the teacher could demonstrate with two hands while the student robot must use a single manipulator. The key idea in the approach is to perform trajectory optimization in the student's action space such that the distance between the final state in the student trajectory and that of the teacher's trajectory is minimized. To make this process tractable when forward simulation is computationally expensive, the authors propose training a forward dynamics model that can be queried instead. The student dataset created through this process can then be input to an off-the-shelf IL or RLfD (reinforcement learning from demonstration) algorithm that produces a proficient student policy. The authors evaluate their method on cloth manipulation and box rearrangement in simulation and demonstrate transfer to real hardware.
review: The authors propose an intuitive recipe for addressing a challenging and important problem in robotics: cross-morphology imitation learning. Progress in this area can unlock significant advances, such as robot learning from videos of humans as well as reusable expert data across a range of robot morphologies. The proposed method is technically sound and achieves strong improvement over baselines in the experiments. Cloth manipulation is a difficult task, and the video attachment demonstrates the proficiency of the single-picker student in accomplishing the task demonstrated by the dual-picker teacher. 

The paper has some weaknesses as well. First, the method is simple: essentially, it simply trains a forward dynamics model then performs the cross-entropy method, which is a well-known formula (e.g., visual foresight [https://arxiv.org/abs/1812.00568]). While the dynamics model is motivated by the use of the expensive cloth simulator, even this component is not required if fast forward simulation is available. Its application to cross-morphology IL is novel as far as I know, but given the large body of work in imitation from observation (IfO) that I am not fully familiar with, I would not be surprised if another reviewer is aware of a similar work that has tried this approach. Second, evaluation is significant but a bit limited. The rearrangement task is very simple as the authors note, and the cloth manipulation involves only two softgym tasks. Also, the real-world evaluation is limited to demonstrating sim2real policy transfer rather than learning from real data. Given that there is no simulator of the real world, it would be interesting if a visual dynamics model learned on real data could be used in the proposed method. Third, the clarity of the paper would benefit from several clarifications and added details, some of which I list below: 

- How is the dataset of random actions for training the dynamics model generated? What is the random action policy, the range of the action deltas, the episode horizon, etc? This would be good to add to the appendix as adequate coverage of the state space is essential for the dynamics model to be useful in MAIL.
- How many particle positions N are used in the state representation?
- Section 3.3 says "we use them with the simulator to obtain the optimized trajectories" -- does D_student consist of states from the dynamics model or the simulator?
- How are demonstrations provided? Is it a scripted policy? What are the details of the scripted policy?
- Figure 3 shows mean and error bars across seeds - are these training seeds or evaluation seeds? If the latter, is there only a single training seed per algorithm?
- The sim-to-real gap appears to be extremely large for the Three Boxes task -- how does an image-based policy trained in sim transfer zero-shot to the real world in this case? Is there a state estimation pipeline in real?
- The authors note an emergent behavior of straightening the cloth out on the plank. Is this behavior captured in D_student or generated by the LfD algorithm?
- The authors note GNS performs poorly because it is trained on small action deltas. Why not just train it on the larger action deltas for a more fair comparison?
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: **Post-Rebuttal Update**: I am keeping my Weak Accept score. The authors have addressed most of my concerns, but I am still not convinced that the method is particularly novel. I still recommend acceptance as the paper has other strengths.

I recommend acceptance contingent on making the paper clearer in the ways I have noted in the review. This should be very doable in the rebuttal period as I haven't explicitly asked for any additional experiments. However, if the other reviewers disagree, especially if they have found highly related work I am not aware of, I may reconsider.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper introduces a framework called Morphological Adaptation in Imitation Learning (MAIL), enabling learning from demonstrations across robots with different morphologies. It addresses the challenge of training a robot with a different number of end-effectors compared to the teacher robot. Using trajectory optimization, the framework converts state-based demonstrations into suboptimal trajectories in the student's morphology. These trajectories are then used by a Learning from Demonstrations (LfD) method to improve the student's performance by interacting with the environment.

The paper's main contributions are the proposal of the MAIL framework that allows training a robot with demonstrations that has a morphological mismatch between end-effectors. The paper demonstrates MAIL's effectiveness on manipulation tasks involving rigid and deformable objects, including 3D cloth manipulation with obstacles. The policy learned using MAIL also transfers successfully to a real robot.
review: The paper demonstrates a high level of quality in terms of its methodology, experimental setup, and results. The authors propose a novel framework, MAIL, and provide a detailed explanation of its components. The experiments are well-designed and thorough, showcasing the effectiveness of the framework on challenging cloth manipulation tasks . The use of both simulated and real-world scenarios adds to the robustness of the study.

Clarity: The paper is generally well-written and effectively conveys the main ideas and contributions of the work. The authors provide clear explanations of the framework and its components, making it accessible to readers. However, some sections could benefit from further clarification, especially regarding specific technical details related to trajectory optimization and the adaptation of LfD methods.

Originality: The paper introduces a novel framework, MAIL, which addresses the problem of learning from demonstrations across robots with different morphologies. While the concept of LfD is not new, the specific focus on morphological adaptation is a novel contribution. The approach of converting state-based demonstrations into suboptimal trajectories and leveraging trajectory optimization for speed and efficiency is also original. The combination of these elements distinguishes the paper from previous works.

Significance: The work presented in the paper has significant implications for the field of imitation learning and robotics. The ability to train robots with different morphologies using demonstrations from other robots expands the possibilities for knowledge transfer and robot learning. The demonstrated improvements over baselines and successful deployment on a real robot highlight the practical relevance of the framework. The generalizability of the approach to different tasks and end-effector transfers further enhances its significance.

Strengths:
1. Introduction of the novel framework MAIL, addressing the problem of morphological adaptation in LfD.
2. Thorough experimental evaluation showcasing the effectiveness of MAIL on challenging manipulation tasks.
3. Successful deployment of the learned policy on a real robot, demonstrating practical applicability.
4. Generalizability of the framework to various instances of end-effector transfer, illustrating its versatility.

Weaknesses:
1. Limited Discussion on Limitations: The paper does not extensively discuss the limitations of the proposed framework. Addressing potential challenges, constraints, or scenarios where the framework might not perform well would provide a more balanced perspective on its applicability. In line 317, where n<=m, the author provided a method to tackle mismatch when the teacher has less action space than the robot. However, such an approach would limit the capability of the robot and seems to be a wasteful approach.
2. Incomplete Technical Explanations: Some technical aspects, such as trajectory optimization and the adaptation of LfD methods, could be further clarified. Improved explanations and additional details would help readers better understand the proposed framework and its underlying mechanisms (as creating the optimized student dataset seems to be a key point in this paper).
3. Limited Exploration of Alternative Optimization Techniques: The paper primarily focuses on trajectory optimization as the means to convert demonstrations. Exploring and discussing alternative optimization techniques or considering the limitations of the chosen approach would enhance the robustness of the framework.
4. Lack of Discussion on Dataset Variability: The paper does not discuss the variability or diversity of the datasets used for training and evaluation. Exploring the impact of dataset characteristics, such as the number of demonstrations, their quality, or the diversity of tasks, would provide deeper insights into the framework's performance.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: I think the paper demonstrates a novel framework and presents valuable contributions in the field of morphological adaptation in imitation learning.

-----------------------------

**Post rebuttal**

The author had addressed some of my concerns on the contribution and the lack of details in the setup. I think this work does have value in domain adaptation for learning from demonstration.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper studies learning from demonstrations that have substantially different morphologies than the current robot, hence the use of the term "cross-morphology". The method is named "Morphological Adaptation in Imitation Learning (MAIL)" and it tries to tackle policy learning for a robot with $m$ end-effectors from demonstrators with $n$ end-effectors, where in general $n \ne m$. Since the end-effector count is not necessarily equal, the key idea of MAIL is not to match morphologies but to instead match the object states instead. The paper tests on robotic cloth manipulation applications including simulation and in the real world, along with a simulated block arrangement task. They show that their method is able to outperform numerous competing baseline methods.

review: Strengths: 

1. Learning from cross-morphology demonstrations is an important problem especially when we consider humans providing demonstrations to robots, due to completely different morphologies.

2. While the paper only tests on two variants of cloth manipulation (as well as a less-interesting simulated block arrangement task), it benchmarks against a lot of other methods, so I appreciate that rigorous experimentation. There are multiple variants of SAC.

3. Cloth manipulation is itself challenging and I appreciate their efforts in doing so.

Areas to potentially improve:

I think I get the rough idea of the method but there are a lot of confusing aspects to the paper which hinder my understanding. Here are my confusions:

- While the method, MAIL, is advertised as an imitation learning method, it is really a combination of imitation learning and reinforcement learning, with the RL part happening later. This could make it harder to apply for robotics tasks that don't have a suitable simulator. Maybe the RL aspect should be emphasized earlier just so that the reader is aware that RL will be needed.

- Similarly in terms of clarification, MAIL specifically focuses on *end-effector count* morphology, such as using 1 picker or 2 picker. This is somewhat different from what I had thought of originally which could have referred to any human demonstration versus robot demonstration (as humans and robots have different morphologies).

- There are two places in the text that suggest cloth manipulation is very slow to simulate, in lines 62-63 ("These can significantly slow down simulation…") and in lines 147-152 ("Robotic simulators have come a long way in advancing fidelity and speed, but [...] making optimization intractable for challenging simulations"). It took me a while to understand what the paper was saying, since I had assumed their method would try to *avoid* any cloth simulation due to "intractability" but the paper quickly says that it will use NVIDIA FleX simulation anyway! After reading carefully, I think the point is that MAIL still uses NVIDIA FleX, it just doesn't use it for "forward dynamics", instead it uses a learned dynamics model which imitates the simulator but is faster to query than the simulator itself. I think that's the reason (please correct me if I am wrong) but I fear the writing is confusing in that regard.

- As a side comment, I have used NVIDIA's cloth simulation in the past and I found the simulation to be quite fast to run, hence I'm surprised that the paper reports a 50X speed-up (162 fps vs 3.4 fps) with the learned dynamics vs the underlying simulator. Since the paper uses CEM for optimization, isn't it possible to parallelize the underlying FleX simulation to get faster data (even if each individual environment runs at the same speed)?

- Lines 152-153: the paper says it uses a CNN-LSTM based forward dynamics model. But then why are the inputs represented as N particle positions? Normally, a CNN would take in an image as input, right? Taking in particles would imply using a graph neural network or a point cloud neural network, where I might interpret a "particle" as a graph node or a single point, respectively. Incidentally, why use CNNs and LSTMs at all if one can use Transformers?

- Sections 3.3 and 3.4: I was getting confused about why the learned dynamics model was used in 3.3, yet the underlying simulator was used in 3.4. Two comments: first, maybe it would help to clearly identify when the learned dynamics versus the underlying simulator is used in the different stages of MAIL. Second, since the underlying simulator is used in 3.4 anyway, that seems to suggest the underlying cloth simulation is fast enough? Just another thought: maybe it helps to identify the rough computational resources needed for both 3.3 and 3.4 to understand which of those two stages is the time bottleneck.

Other concerns have to do with some possible over-claiming:

- Line 72 says that "MAIL can handle general instances of $n$-to-$m$ end-effector transfer. However this is not what is being shown. Instead, as line 74 says, the specific example MAIL shows is handling 3-to-2, 3-to-1, and 2-to-1 end-effector transfer. Saying "general instances" implies that we can have $n$ and $m$ be arbitrary integers. Thus I would suggest changing line 72 accordingly.

- Line 318 to 319: similarly, the paper says "Thus, MAIL is capable of general $n$-to-$m$ end-effector transfer. This is not what was shown. Also just before this, line 316, I would change "Thus, MAIL can solve a task using…" to "Thus, MAIL can solve this particular task using…" because this is focusing on the particular task of block rearrangement.

- Line 368 (in the conclusion), similarly, the paper says "We further showed LfD generalizability to any transfer from $n$-to-$m$ end-effectors." I suggest rewriting the sentence to say "3-to-2, 3-to-1, and 2-to-1" because that is what was actually shown in experiments, and it should condition on the specific tasks that were shown.

Finally, one comment has to do with related work. I would recommend adding more material about cross-morphology learning. For example, cross-embodied IRL https://x-irl.github.io/ and AVID https://arxiv.org/abs/1912.04443 which tries to translate demonstrations from humans to robots. These are just two examples I am aware of; there are likely many more especially if we count imitation from dexterous manipulators (e.g,. Human hand vs Allegro hand). I would probably de-emphasize the generic "Imitation from Observation" portion unless those specifically deal with the challenges of translating from human demonstrations to robot demonstrations.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This method and paper has some promise. There are a few concerns I have as noted in the review about how much this will generalize to $m$ and $n$ end-effectors as well as some confusing aspects of the text. 

Update: based on the author response where they said they would modify the writing of the paper about the "any n-to-m transfer" part, I'm OK with a weak accept.
review_title: null

Meta-review: This paper introduced a cross-morphology imitation learning method, which aims to learn from demonstrations where the teacher robot's morphology is significantly different from the student's, e.g., different numbers of end-effectors. The proposed approach leveraged a learned forward dynamics model to perform the cross-entropy method such that the distance between the final state in the student's trajectory and that of the teacher's trajectory is minimized. Evaluations were conducted on cloth manipulation and box rearrangement tasks in simulation, and sim2real results have been shown on a physical robot arm.

This paper received positive ratings from four reviewers, who gave Weak Accept after the rebuttal. However, there was a lack of enthusiastic support. Some lingering issues were brought up by the reviewers, including the limited novelty of the proposed method and the experiments on toy problems, which are also acknowledged by the authors to some extent. Toward the end of the discussion period, Reviewer ExJj expressed that "Inclined to reject this paper since authors have been unable to convince me that their approach isn't just a trivial combination of existing methods." Reviewer igC1 "will keep it as a weak accept as I'm not convinced the method is very novel."

The AC read the paper after going through the reviews, rebuttal, and post-rebuttal discussions in detail. While there is ample room to improve this work, it has offered a reasonable solution to a well-motivated problem. To encourage this line of work, the AC would recommend this paper to be accepted, but with a low confidence, and leave it to the PCs for the final decision.
Decision: Accept (Poster)