Learning Robot Manipulation from Cross-Morphology Demonstration
Authors: Gautam Salhotra
I-Chun Arthur Liu
Gaurav S. Sukhatme
Venue: CoRL 2023 Poster (2023)
Keywords: Imitation from Observation
Learning from Demonstration

Abstract: Some Learning from Demonstrations (LfD) methods handle small mismatches in the action spaces of the teacher and student. Here we address the casewhere the teacher’s morphology is substantially different from that of the student. Our framework, Morphological Adaptation in Imitation Learning (MAIL), bridges this gap allowing us to train an agent from demonstrations by other agents with significantly different morphologies. MAIL learns from suboptimal demonstrations, so long as they provide some guidance towards a desired solution. We demonstrate MAIL on manipulation tasks with rigid and deformable objects including 3D cloth manipulation interacting with rigid obstacles. We train a visual control policy for a robot with one end-effector using demonstrations from a simulated agent with two end-effectors. MAIL shows up to 24% improvement in a normalized performance metric over LfD and non-LfD baselines. It is deployed to a real Franka Panda robot, handles multiple variations in properties for objects (size, rotation, translation), and cloth-specific properties (color, thickness, size, material).

Learning Robot Manipulation from Cross-Morphology Demonstration
Gautam Salhotra" I-Chun Arthur Liu" Gaurav S. Sukhatmel
Robotic Embedded Systems Laboratory
University of Southern California
[salhotra, ichunliu, gaurav] @usc.
Abstract: Some Learning from Demonstrations (LID) methods handle small mis- matches in the action spaces of the teacher and student. Here we address the case where the teacher's morphology is substantially different from that of the stu- dent. Our framework, Morphological Adaptation in Imitation Learning (MAIL), bridges this gap allowing to train an agent from demonstrations by other agents with significantly different morphologies. MAIL learns from suboptimal demon- strations, so long they provide some guidance towards a desired solution. We demonstrate MAIL on manipulation tasks with rigid and deformable objects in- cluding 3D cloth manipulation interacting with rigid obstacles. We train visual control policy for a robot with one end-effector using demonstrations from sim- ulated agent with two end-effectors. MAIL shows up to 24% improvement in a normalized performance metric over LfD and non-LiD baselines. It is deployed real Franka Panda robot, handles multiple variations in properties for objects (size, rotation, translation), and cloth-specific properties (color, thickness, size, material), An overview is on this website.
Keywords: Imitation from Observation, Learning from Demonstration
Introduction
Learning from Demonstration (LfD) [1, 2] is set of supervised learning methods where teacher (often, but not always, human) demonstrates a task, and a student (usually robot) uses this information to learn to perform the same task. Some LiD methods cope with small morphological mismatches between the teacher and student [3, 4] (e.g., five-fingered hand to two-fingered gripper). However, they typically fail for a large mismatch (e.g., bimanual human demonstration to a robot arm with one gripper). The key difference that reproduce the transition from a demonstration state the next, no single student action suffices a sequence of actions may be needed.
Supervised methods are appealing where demonstration free methods [5] do not converge or under- perform [6] and purely analytical approaches are computationally infeasible [7, 8]. In such settings, human demonstrations of complex tasks are often readily available egg., is straightforward for a human to show a robot how to fold a cloth. An LiD-based imitation learning approach is appealing in such settings provided we allow the human demonstrator use their body the way they find most convenient (e.g., using hands to hang cloth clothesline to dry). This requirement induces potentially large morphology mismatch want to learn and execute complex tasks with deformable objects single manipulator robot using natural human demonstrations.
We propose a framework, Morphological Adaptation in Imitation Learning (MAIL), to bridge this mismatch. We focus on cases where the number of end-effectors different from teacher to student,
Equaa contribution
1 G.S. Sukhatme holds concurrent appointments as Professor at USC and as an Amazon Scholar. This paper describes work performed at USC and is not associated with Amazon.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

although the method may be extended to other forms of morphological differences. MAIL enables policy learning for a rohot with m end-effectors from teachers with n end-effectors. It does not require demonstrator actions, only the states of the objects in the environment making it potentially useful for variety of end-effectors (pickers, suction gripper, two-fingered grippers, or even hands). uses trajectory optimization to convert state-based demonstrations into (suboptimal) trajectories in the student's morphology. The optimization accuracy for speed, especially useful for tasks with high-dimensional state and observation spaces. The trajectories are then used by an Lff) method, optionally with exploration components like re- inforcement learning, which is adapted to work with sub-optimal demonstrations and improve upon them by interacting with the environment.
learned (forward) dynamics model to trade
Though the original demonstrations contain states, generalize the solu- tion to work with image observations in the final policy. We showcase our method on challenging cloth manipula- tion tasks (Sec. 4.1) for a robot with one end-effector, using image observations, shown in Fig. This setting chal- lenging for multiple reasons. First, cloth manipulation is easy for bimanual hu- man demonstrators but challenging for a one-handed agent (even humans find cloth manipulation non-trivial with one hand). Second, deformable objects exist a continuous state space; image ob- servations in this setting are also high- dimensional. Third, the cloth being nipulated makes a large number of con- tacts (hundreds) that are made/broken per time step. These can significantly slow down simulation, and consequently learning and optimization. We make the following contributions:
Leaetee
Spat temponal
Indirect
Trajectory
Dynamics Mocel
Optiiiatioo
effectors
demos
HD Merhhd
serd-elllctooe (1=2) der demonsstaations
sn enc--ellectors (row)) rollouts in simulation and real world
Figure I: MAIL generalizes LID to large morphological mismatches between teacher and student in difficult ma- nipulation tasks. We show an example task: hang cloth to dry plank (DRY CLOTH). The demonstrations bimanual, yet the robot learns to execute the task with a single arm and gripper. The learned policy transfers to the real world and is robust to object variations.
We propose novel framework, MAIL, that bridges the large morphological mismatch in LfD. MAIL trains robot with end-effectors learn manipulation from demonstrations with different (n) number of end-effectors.
We demonstrate MAIL on challenging cloth manipulation tasks on robot with one end-effector. Our tasks have a high-dimensional (> 15000) state space, with several 100 contacts being made/broken per step, and are non-trivial to solve with one end-effector. Our learned agent out- performs baselines by up to 24% normalized performance metric and transfers zero-shot to variant of 3D cloth manipulation with obstacles DRY CLOTH. 3. We illustrate MAIL providing different instances of end-efficcor transfer, such as a 3-to-2, 3- to-1, and 2-to-1 end-effector transfer, using simple rearrangement task with three rigid bodies in simulation and the real world. We further explain how MAIL can potentially handle more instances of n-to-m end-effector transfer.
real robot. We introduce
2 Related Work
Imitation Learning and Reinforcement Learning with Demonstrations (RLfD): Imitation learn- ing methods [9, 10, 11, 12, 13] and methods that combine reinforcement learning and demonstra- tions [14, 15, 1, 2] have shown excellent results in learning a mapping between observations and ac- tions from demonstrations. However, their objective function requires access to the demonstrator's ground truth actions for optimization. This infeasible for cross-morphology transfer due action space mismatch. To work around this, prior works have proposed systems for teachers provide demonstrations in the students' morphology [16] which limits the ability of teachers to efficiently provide data. Similar to imitation learning, offline RL [17, 18, 19] learns from demonstrations stored

---

in dataset without online environment interactions. While offline RL can work with large datasets of diverse rollouts to produce generalizable policies [20, 21], it requires the availability of rollouts that have the same action space as the learning agent. MAIL learns across morphologies and is not affected by this limitation.
Imitation from Observation: Imitation from observation (lfO) methods [3, 9, 22, 23, 24, 25, 26] learn from the states of the demonstration; they do not use state-action pairs. In [27], an approach is proposed to learn repetitive actions using Dynamic Movement Primitives [28] and Bayesian op- timization to maximize the similarity hetween human demonstrations and rohot actions. Many IfO methods [3, 23, 24, 29] assume that the student can take a single action to transition from the demon- stration's 's current state to the next state. Some methods [3, 23] use this to train an inverse dynamics model infer actions. Others extract keypoints from the observations and compute actions by sub- tracting consecutive keypoint vectors. XIRL [30] uses temporal cycle consistency between demon- strations to learn task progress as a reward function, which is then fed to RL methods. However, when the student has a different action space than the teacher, it may require more than one action for the student to reach consecutive demonstration states. For example, in an object rearrangement task, a two-picker teacher agent can move two objects with one pick-place action. But a one-picker student will need two or more actions to achieve the same result. Zero-shot visual imitation [9] assumes that the statistics of visual observations and agents observations will be similar. However, when solving task with different numbers of arms, some intermediate states will not be seen in teacher demonstrations. State-of-the-art learning from observation methods [25, 31] have made significant advancements in exploiting information between states. However, their tasks have much longer horizons, hence more states and learning signals than ours. Whether these methods work well on short- horizon,, difficult manipulation tasks is uncertain. To address this and provide a meaningful comparison, we conducted experiments to compare MAIL with these methods (Sec. 4).
Trajectory optimization algorithms [32, 8, 33] optimize trajectory by minimizing cost function, subject to a set of constraints. It has been used for manipulation of rigid and deformable objects [7], even through contact [34] using complementarity constraints [35]. Indirect trajectory optimization only optimizes the actions of a trajectory and uses a simulator for the dynamics instead of adding dynamics constraints at every step.
Trajectory Optimization:
Learning dynamics models is useful when there is simulator, if the simulator is too slow or too inaccurate. Learned models have been used with Medel-Predictive Control (MPC) to speed up prediction times [36, 37, 38]. A common use case is model- RL [39], where learning the dynamics is part of the algorithm and has been shown to learn dynamics from states and pixels [40] and applied to real-world tasks [41].
Learned Dynamies:
3 Formulation and Approach
3.1 Preliminaries
We formulate the problem as partially observable Markov Decision Process (POMDP) with state 8 E S, action a E A, observation o E O, transition function T S x A + S, horizon H, discount factor y and reward function r : S A + R. The discounted return at time t is Rg = D fr((s, a.) and 8; T(s;-1, a;) task is instantiated with a variant sampled from the task distribution, a V. The initial environment state depends on the task variant, so su(v). e V.
We train policy me to maximize expected reward J(TT)) of an episode over task variants v. J [ Eowy[Ho), subject to initial state sv(v) and the dynamics from T.
For an agent with morphology M, we differentiate between datasets available as demonstrations (DM..) and those that are optimized (Doptim). For our cloth environments, our teacher mor- phology is two-pickers (M = 2p) and student morphology is one-picker (M = 1p). We assume the demonstrations are from teachers with morphology that can be different from the student (and from each other). We refers to these teacher demonstrations, Deaahhrs emphasize that they do not necessarily come from expert or oracle. Further, these can be suboptimal. The demonstrations
3

---

Pardom Actiann end-effectors Dissas (1s.a, ....))
Optimized
Demos
and-affactors
Indrect Trajectory Optimization
Massin (a
Lemmed Dynamics I| aPudla
1yea Sulls
Tencher Damon eftectteccors Draderr (0..%.....
HD Mathod
An example cloth folding task with demonstrations from teacher with n = end- effectors, deployed on Franka Panda with network to predict the forward dynamics of the object being manipulated in simulation, using random action dataset D Random. For every state transition, we match the predicted particle dis- placements from our model, APpred to that of the simulator, APasm. Given this learned dynamics and the teacher demonstrations we use indirect trajectory optimization to find student actions that solve the task. The optimization objective is to match with the object states in the demonstration. Finally, we pass the optimized dataset D Student to a downstream LfD method to get a final policy that generalizes to task variations and extends task learning to image space, enabling real-world deployment.
Figure 2:
end-effector (parallel-jaw gripper). We train
are state trajectories TL = (80,.... 8H -1) The teacher dataset is made up of KT such trajectories, D Deeacher = Vir 1,..., Kr. using a few task variations from the task distribution Va ~ V.
We now discuss the components of MAIL, shown in Fig. 2. The user provides teacher demonstra- tions D, Teacher- First, create dataset of random actions, D Random and use it train dynamics model, Tws Top reduces computational cost when dealing with contact-rich simulations like cloth ma- nipulation (Sec. 4.1). Next, we convert each teacher demonstration to trajectory suitable for the student's s morphology. For our tasks, we find gradient-free indirect trajectory optimization [33] per- forms the best (Appendix Sec. A.1). We used To for this optimization as provides the appropriate speed-accuracy trade-off. The optimization objective is to match with object states in the demonstra- tion (we cannot match demonstration actions across morphologies). We combine these optimized trajectories to create dataset D Student for the student. Finally, pass D Student to downstream L.fD method to learn policy that generalizes from the task variations in 'D Deachher to the task distribution V. It also extends π to use image observations and deploys pero-shot on a real robot (rollouts in Fig. 5).
3.2 Learned Spatio-temporal Dynamics Model
MAIL uses trajectory optimization to convert demonstrations into (suboptimal) trajectories in the student's morphology. This can be prohibitively slow for large state spaces and complex tasks such as cloth manipulation. Robotic simulators have come long way in advancing fidelity and speed, but simulating complex deformable objects and contact-rich manipulation still requires significant com- putation making optimization intractable for challenging simulations. We use NVIDIA FLeX simulator that based on extended position- dynamics [42]. We learn a CNN-LSTM based spatio-temporal forward dynamics model with parameters vh, Ty. approximate cloth dynamies, T. This offers speed-accurace accuracy trade-off with tractable computation time in environments with large state spaces and complex dynamics. The states of objects are represented as N particle positions: P = {p: 1 Each particle state consists of its and z coordinates. For each task, we generate a corpus of random pick-and actions and store them the dataset DD Random = )d,), where i = 1,... KR and d, = (P,, a, P.). For each datum i, we feed P. to the CNN network to extract features of particle connectivity. These features are concatenated with a; and input to the LSTM model extract features based on the previous partiele positions. fully connected layer followed by layer normalization and tanh activation is used to learn the non-linear combinations of features. The outputs the predicted particle displacements. The objective function is the

---

distance between predicted and ground -ruth particle displacements, || P Here A Psim = (Apt)=s,..... is obtained from the simulator and Ap; Pi+1 P; for every partiele i.
Due to its simplicity, the CNN-LSTM dynamics model provides fast inference, compared to a sim- ulator which may have perform many collision checks at any time step. This speedup is crucial when optimizing over large state space, as long as the emrors in particle positions are tolerable. In our experiments, we were able to get 162 fps with Twp compared to 3.4 fps with the FleX simulator, a 50x speed up (Fig. 8). However, this stage is optional if the environment is oiw-dimenioonl,, or the simulation speed-up from inference is not significant. Simulation accuracy is important when training a final policy, to provide accurate pick-place locations for execution on a real robot. Hence, the learned dynamics model not used for training in the downstream LfD method.
3.3 Indirect Trajectory Optimization with Learned Dynamics
We use indirect trajectory optimization [33] to find the open-loop action trajectory to match the teacher state trajectory. TT. This optimizes for the student's 's actions while propagating the state with a simulator. We use the learned dynamics To to give us fast, approximate optimized trajectories. This is in contrast to direct trajectory optimization (or collocation) that optimizes both states and actions at every time step. Direct trajectory optimization requires dynamics constraints to ensure consistency among states being optimized, which can be challenging for discontinuous dynamics. We use the Cross-Entropy Method (CEM) for optimization, and compare this against other methods, such SAC (Appendix A.1). Optimization hyperparameters are described in 5. The optimization objective is to match the object's goal state $gool in the demonstration with the same task variant V.-- Formally, the problem is defined as:
min | ggoal lælle subject to &u = 80(2)) and 88+1 T(s oz) VL 0, H (1)
where 811 is the predicted final state. Note that if TT has longer time horizon, it would help to match intermediate states and use multiple-shooting methods. After optimizing the action trajec- tories for each demonstration TT, E Deechherr we use them with the simulator to obtain the opti- mized trajectories in the student's 's morphology. These are combined to create the student dataset,
D Pttudent = {Ti, 72, 73.... J. where 5; (st, Ot, at, S1+1, 0++1, Tt, djvt 1... H For gen- eralizability and real-world capabilities, we train LfD method using D Studeett Note that we use the learned dynamics model at this stage, trading faster simulation speed for lower accuracy in the learned model. This is also partially responsible for why Dstudent contains suboptimal demon- strations. To reduce the effect of learned model errors, once we obtain the optimized actions, we perform a rollout with the true simulator to get the demonstration data.
3.4 Learning from the Optimized Dataset
Our chosen LfD method is DMfD [43], an off- off-policy RLfD actor-critic method that utilized expert demonstrations well rollouts from its own exploration. It learns using an advantage-weighted the simulator instead of the learned dynamics model Tp at this stage, because accuracy is important in the final reactive policy. Hence, we cannot take the speed-accuracy tradeoff that To provides. However, one may choose to use other LfD) methods that do not need to interact with the environ- ment [45], in which case neither simulator nor learned dynamics are needed.
formulation [44] balanced with exploration component [5]. As mentioned above,
As part of tuning, we employ 100 demonstrations, about orders of magnitude fewer than the 8000 recommended by the original work. To prevent the policy from overfitting suboptimal demonstrations in Studentt we disable demonstration-starl state matching, i.e., resetting the agent to demonstration states and applying imitation reward (see Appendix A.5). These were originally proposed [46] as reference state initialization (RSI). These modifications are essential for our Lfl) implementation, where the demonstrations do not come from an expert.
From DMfD, the policy is parameterized by parameters 8, and learns from data collected in replay buffer B. The policy loss contains ddvaantage-weighie loss CA where actions are weighted
5

---

by the advantage function (s) = (* (*(s) and temperature parameter A. It also contains an entropy component Lg to promote exploration during data collection. The final policy loss - is a combination these terms (Eq. 2).
(a (2
A = E 8,0,0~8
log To exp
LE E log -((s) 8,0,0~
C = (1- + WELE, 05 WE 3
(2)
where we is tuneable hyper-parameter. The resulting policy is denoted as TT. We pre-populate buffer B with D Student Using LfD, we extend from state inputs to image observations, and gener- alize from va to any variation sampled from V.
Experiments
Our experiments are designed answer the following: (1) How does MAIL compare to state-of-the- art (SOTA) methods? (Sec. 4.2) (2) How well can MAIL solve tasks in the real world? (Sec. 4.2.1) (3) Does MAIL generalize to different n-to-m end-effector transfers? (Sec. 4.3) Additional periments demonstrating how different MAIL components affect performance are in Appendix A.
4.1 Tasks
We experiment with cloth manipulation tasks that are easy for humans to demonstrate but difficult to perform on a robot. We also discuss a simpler rearrangement task with rigid bodies to illustrate generalizability. The tasks are shown in Appendix Fig. We choose a 6-dimensional pick-and-plas action space, with xyz positions for pick and place. The end-effectors are pickers simulation, and two-finger parallel jaw gripper the real robot.
CLOTH FOLD:
Fold square cloth in half, along specified line. DRY CLOTH: Pick up square
cloth from the ground and hang it on a plank to dry, variant of [47]. THREE BOXES: A simple environment with three boxes along line that needs to rearranged to designated goal locations. For details on metrics and task variants, see Appendix B.
We use particle positions the state for train- ing dynamics models and trajectory optimiza- tion. We use a 32x32 RGB image as the vi- sual observation, where applicable. We record pre-programmed demonstrations for teacher dataset for each task. Details of the datasets to train the LID method and the dynamics model in Appendix E and Appendix F. The in- stantaneous reward, used in learning the policy, is the task performance metric at a given state. Further details on architecture and training are in the supplementary material. In all experi- ments, we compare each method's normalized performance, measured the end of the task given by p(t) p(a,)-p(ss)
GV5
SAC-CUIL
0.8
SAC-DrQ
GP6
SAC-DrQ
MAL [oand)
Nornakeo Pelsmers
100
Chorre
CLOTE
Figure 3: SOTA performance comparisons. For each training run, used the best model in each seed's training run, and evaluated using 100 roll- outs across 5 seeds, different from the training seed. Bar height denotes the mean, error bars in- dicate the standard deviation. MAIL outperforms all baselines, in some cases by as much as 24%.
Popt Popt-plap -P(o))' where p is the per- formance metric of state at time t, and Pope the best performance achievable by the task. We use p(ff) at the end of the episode (t = H).
4.2 SOTA comparisons
Many LiD baselines (Sec. 2) are not directly applicable, as they do not handle large differences in action space due to different morphologies. We compare MAIL with those LfD baselines that produce policy with image observations, given demonstrations without actions.
1. SAC-CURL [48]: An image-based RL algorithm that uses contrastive learning and SAC [5] as the underlying RL algorithm. It does not require demonstrations.

---

2. SAC-DrQ [49]: An image-based RL algorithm that uses regularized Q-function, data augmen- tations, and SAC as the underlying RI. algorithm. It does not require demonstrations.
3. GNS [50]: A SOTA method that represents cloth as a graph and predicts dynamics using graph neural network (GNN). does not require demonstrations but learns dynamics from the random actions dataset. We run this learned model with a planner [51], given full state information.
SAC-DiQ-IR: custom variant of SAC [5] that uses DrQ-based [49] image encoding and a state- only imitation reward (IR) to reach the desired state of the object to be manipulated. does not imitate actions, they unavailable.
5. GAIfO [25]: An adversarial imitation learning algorithm that trains a discriminator on state-state pairs (s, s') from both the demonstrator and agent. This is popular extension of GAIL [13] that learns the same from state-action pairs (s, a..
6. GPIL [31] A goal-directed LfD method that uses demonstrations and agent interactions to learn goal proximity function. This function provides dense reward to train policy.
Fig. 3 has performance comparisons against all baselines. In each environment, the first three columns are demonstration-fiee baselines, and the last four are LID methods. MAIL outperforms all baselines, in some cases by as much as 24%. For the easier CLOTH FOLD task, the SAC-DrQ baseline came within 11% of MAIL.
However, all baselines do not perform well in the more difficult DRY CLOTH task. RL meth- ods fail because they have not explored the pa- rameter space enough without guidance from demonstrations. Our custom LiD baseline, SAC-DiQ-IR, has reasonable performance, but
(a) Teacher demonstration with three pickers.
the results show that naive imitation alone not good form of guidance to solve it. The other LID basclines, GAIFO and GPIL, have poor performance in both environments. The primary reason is the effect of cross- morphological demonstrations. They perform significantly better with student morphology demonstrations, even if they are suboptimal. Moreover, environment difficulty also plays an important part in the final performance. These and other ablations in Appendix A.
(b) Final policy: Two pickers
(c) Final policy: One picker
(d) Final policy: One Franka Panda robot
Sample trajectories of the THREE BOXES task. A three-picker teacher trajectory to reach the goal state (Fig. 4a). Final policies of the two-picker and one-picker agent, and real-world execution of the one-picker agent.
Figure 4:
Surprisingly, the GNS baseline with structured dynamics does not perform well, even though has been used for cloth modeling [52]. This is because it designed learn particle dynamics via small displacements, but our pick-and and-place action space enables large displacements. Similar [51], break down each pick-and-place action into 100 delta actions to work with the small displacements that GNS is trained planning will accumulate emror from the 100 GNS steps for every action of the planner, which can grow superlincarly due to compounding errors. This makes it difficult to solve the task. It is especially seen in DRY CLOTH, where the displacements required to move the entire cloth over the plank are much higher than the displacements needed for CLOTH FOLD. The rollouts MAIL on DRY CLOOH show the agent following the demonstrated guidance it learned to hang the cloth over the plank. also displayed an emergent behavior to straighten the cloth on the plank to spread it out and achieve higher performance. This was not seen in the two-picker teacher demonstrations.
Thus,
4.2.1 Real-world results
For DRY CLOTH and CLOTH FOLD tasks, we deploy the learned policies on Franka Panda robot with a single parallel-jaw gripper (Fig. 5, statistics over 10 rollouts). We test the policies with many different variations of square cloth (size, rotation, translation, thickness, color, and material). See Appendix D for performance metrics. The policies achieve ~ 80% performance, close to the average simulation performance, for both tasks.
7

---

4.3 Generalizability
We show examples of how MAIL learns from a demonstrator with different number of end- effectors, in a simple THREE BOXES task (Fig. 4). Consider a three-picker agent that solves the task in one pick-place action. Given teacher demonstrations Teacher: we transfer them into one- picker or two-picker demonstrations using indirect trajectory optimization and the learned dynamics model. These are the optimized datasets that are fed to a downstream LfD method. In both cases, the LfD method learns a model, specific to that morphology, to solve the task. It generalizes from state inputs in the demonstrations to the image inputs received from the environment. Fig. 4 shows the three picker demonstration, a 3-to-2 and 3-to-1 end-effector transfer. We have also done this for the 2-to-1 case (omitted here for brevity). These examples illustrate n-to-m end-effector transfer with n m; it is trivial to perform the transfer for n-to-m with I m by simply appending the teacher's action space with m arms that do no operations.
4.4 Limitations
MAIL requires object states in demonstrations and during simulation training, however, full state information is not needed at deployment time. It has been tested on the pick-place action space. It has been tested only on cases where the number of end-effectors is different from teacher to stu- dent. While it works for feeeffreeeency actions (Appendix A.7), it will likely be difficult to optimize actions to create the student dataset for high-dimensional actions. This is because the curse of di- mensionality will apply for larger action spaces when optimizing for Patadent- The state-visitation distribution of demonstration trajectories must overlap with that of the student agent; this overlap must contain the equilibrium states of the demonstration. For example, one-gripper agent cannot reach a demonstration state where two objects are moving simultaneously, but it can reach a state where both objects are stable at their goal locations (equilibrium), MAIL cannot work when the student robot unable to reach the goal or intermediate states in the demonstration. For example, in trying to open a flimsy bag with two handles, both end-effectors may simultaneously be needed to keep the bag open. When we discuss generalizability for the case S m, our chosen method tackle morphological mismatch is to use fewer arms on the student robot, in lieu of trajectory optimization. This is inefficient approach since we ignore some arms of the student robot. MAIL builds a separate policy for each student robot morphology and each task. While it is possible to train& a multi-task policy conditioned on a given task (provided as an embedding or a natural lan- guage instruction), extending MAIL to output policies for a variable number of end-effectors would require more careful considcration. Subsequent work could learn a single policy conditioned on the desired morphology another way to think about a base model for generalized LfD.
5 Conclusion
We presented MAIL, a framework that enables LfD across morphologies. Our framework enables learn- ing from demonstrations where the number of end- effectors is different from teacher to student. This enables teachers record demonstrations in the set- ting of their own morphology, and vastly expands the set of demonstrations to learn from. We show an improvement of up to 24% over SOTA baselines and discuss other baselines that are unable to handle a large mismatch between teacher and student. Our experiments are on challenging household cloth ma- nipulation tasks performed by robot with one end-effector based bimanual demonstrations. We showed that our policy can be deployed zero-shot on real Franka Panda robot, and generalizes across cloths of varying size, color, material, thickness, and robustness to cloth rotation and trans- lation. We further showed examples of LID generalizability with instances of transfer from n-to-m end-effectors, with multiple rigid objects. We believe that this is an important step towards allowing LfD to train a robot to learn from any robot demonstrations, regardless of robot morphology, expert knowledge, or the medium of demonstration.
Bry del
Figure 5:
Real-world results for CL.OTH FOLD and DRY CLOTH.
8

---

References
[1] K. Pertsch, Y. Lee, Y. Wu, and J. J. Lim. Demonstration-guiel reinforcement learning with learned skills. 5th Conference on Robot Learning, 2021.
[2] I.-C. A. Liu, S. Uppal, G. S. Sukhatme, J. J. Lim, P. Englert, and Y. Lee. Distilling motion planner augmented policies into visual control policies for robot manipulation. In A. Faust, D. Hsu, and G. Neumann, editors, Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 641-650. PMLR, 08-11 Nov 2022. URL https: pttp//eiecceenn......../.. mlr. www..mll....s...... www.pppessssssieee liu22b. html.
[3] I. Radosavovic, X. Wang, L.. Pinto, and J. Malik. State-only imitation learning for dexterous manipulation. In 2021 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 7865-7871 IEEE, 2021.
[4] Y. Yang, Y. Li, C. Fermuller, and Y. Aloimonos. Robot learning manipulation action plans by "watching" unconstrained videos from the world wide web. Proceedings of the AAAI Conference on Artificial Intelligence, 29(1), Mar. 2015. doi: 100.111099aaii/. v29i1.9671. URL https: /ojs. aajjsaaas aaai. org/ index. http://AA///rrtice//eeeeee
[5] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[6] Hietala, D. Blanco-Mulers G. Alcan, and Kyrki. Learning visual feedhack control for dynamic cloth folding. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1455-1462, 2022. 11010000000100017112200009999990
[7] S. Jin, D. Romeres, A. Ragunathan, D. K. Jha, and M. Tomizuka. Trajectory optimization for manipulation of deformable objects: Assembly of belt drive units. In 2021 IEEE International Conference Robotics and Automation (ICRA), pages 10002-10008. IEEE, 2021.
[8] J. M. Bern, P. Banzet, R. Poranne, and S. Coros. Trajectory optimization for cable-driven soft robot locomotion. In Robotics: Science and Systems, volume 1, 2019.
[9] D. Pathak, P. Mahmoudieh, G. Luo, P. Agrawal, D. Chen, Y. Shentu, E. Shelhamer, J. Malik, A. A. Efros, and T. Darrell. Zero-shot visual imitation. In ICLR, 2018.
[10] C. Finn, T. Yu, T. Zhang, P. Abbeel, and S. Levine. One-shot visual imitation learning via meta- learning. In S. Levine, V. Vanhoucke, and K. Goldberg, editors, Proceedings of the 1st Annual Conference on Robot Learning, volume 78 of Proceedings of Machine Learning Research, pages 357-368. PMLR, 13-15 Nov 2017. URL https: http:///po//////ee//e/e/////tttt mlr., wwr.mrrreee. presss 078/Einn17a.htmm
[11] Y. Duan, M. Andrychowicz. B. Stadie, J. Ho, Schneider, I. Sutskever, Abbeel, and W. Zaremba. One-shot imitation learning. In Proceedings of the 31 International Conference on Neural Information Processing Systems, NIPS' 17, page 1087-1098, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
[12] M. Laskey, J. Lee. R. Fox, A. D. Dragan, and K. Goldberg. DART: noise injection for robust imitation learning. In Ist Annual Conference on Robot Learning, CoRL 2017, Mountain View, California, USA, November 13-15, 2017, Proceedings, volume 78 of Proceedings of Machine Learning Research, pages 143-156. PMLR, 2017. URL http: http:://pocceeeneeee malgg.mml.. mmr. press/v78/ htt///aakellaaa. preas//78/Laskeyy2222 html.
[13] J. Ho and S. Ermon. Generative adversarial imitation learning. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL https://proceeding
httpr/pss.ceppppee//1//////////e pdf.

---

[14] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demon- strations. In Proceedings of Robotics: Science and Systems (RSS), 2018.
[15] M. Vecerík, T. Hester, Scholz, Wang, O. Pietquin, B. Piot, N. M. O. Heess, Rothörl, Lampe, and M. A. Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. ArXiv,
0000000817, 2017.
[16] A. Mandlekar, Booher, M. Spero, A. Tung, A. Gupta, Y. Zhu, A. Garg, S. Savarese, and
L. Fei-Fei. Scaling robot supervision to hundreds of hours with roboturk: Robotic manipulation dataset through human reasoning and dexterity. In 2019 JEEF/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1048-1055. IEEE, 2019.
[17] S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv e-prints, pages arXiv-2005, 2020.
[18] S. Lange, Gabel, and M. Riedmiller. Batch reinforcement learning. In Reinforcement learn- ing, pages 45-73. Springer, 2012.
[19] Fuchs, Song, E. Kaufmann, D. Scaramuzza, and P. Dürr. Super-human performance in gran turismo sport using deep reinforcement learning. IEEE Robotics and Aulomation Letters, 6(3):4257-4264, 2021.
[20] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforce- ment learning, Advances in Neural Information Processing Systems, 33:1179-1199 2020.
[21] P. Rashidinejad, B. Zhu, C. Ma, J. Jiao, and S. Russell. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems, 2021.
[22] Torabi, G. Warnell, and Stone. Recent advances in imitation learning from observation. In Proceedings of the Twenty-Eight International Joint Conference on Artificial Intelligence, IJCAI-19 19, pages nization, 7 2019. doi: 200224966/jaaii2119//8833 URL https:: //o::.////o../ htt./oii.ogrg/g/o org/100 110. 24963/ ijcai. 2019/882.
International Joint Conferences Artificial Intelligence Orga-
[23] F. Torabi, G. Warnell, and P. Stone. Behavioral cloning from observation. In Proceed- ings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI- 18, pages -950-4957. International Joint Conferences on Artificial Intelligence Organization, 2018. doi: 1o016...////iic 2018/687.
URL https: http///ii.og///////////////////t 249.2249666 24963 jjaci.
[24] Y.TT. A. Sun, H.-C. Lin, P.-Y. Wu, and J.-T. Huang. Learning by watching via key- point extraction and imitation learning. Machines, 10(11), 2022. ISSN 2075-1702 doi: 2M.3990sun2KKKyooiiEExtttii URL https://////.. /www///wwwwww///////////// mdpi wttp://///////////0/0.. 11/1049.
[25] F. Torabi, G. Wamell, and P. Stone. Generative adversarial imitation from observation. arXiv preprint IrXivv1807.061 2018.
[26] L. Smith, N. Dhawan, M. Zhang, P. Abbeel, and S. Levine. AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos. In Proceedings of Robotics: Science and Systems, Corvalis, Oregon, USA, July 2020.
[27] Yang, Zhang, C. Settle, A. Rai, R. Antonova, and Bohg. Learning periodic tasks from human demonstrations, IEEE International Conference on Robotics and Automation (ICRA), 2022.
10

---

[28] A. J. Ijspeert, J. Nakanishi, and S. Schaal. Learning rhythmic movements by demonstration using nonlinear oscillators. IEEE/RSJ International Conference on Intelligent Robots and Systems, volume I, pages 958-963, 2002. doi:10. 10..110//RRDDS.. 100000//DDDD222200111111111
[29] F. AI-Hafez, D. Tateo, O. Arenz, G. Zhao, and J. Peters. Ls-iq: Implicit reward regularization for inverse reinforcement learning. In Eleventh International Conference Learning Repre- sentations (ICIR), 2023. URL https: /wpppoeeeiiieeeeee net net/ppff http:/ppdfid=ooo00poo d=o (a444) jg
[30] K. Zakka, A. Zeng, Florence, J. Tompson, J. Bohg, and D. Dwibedi. Xirl: Cross-embodimen inverse reinforcement learning. Conference on Robot Learning (CoRL., 2021.
[31] Y. Lee, A. Szot, S.-H. Sun, and J. J. Lim. Generalizable imitation learning from observation via inferring goal proximity. In Advances in Neural Information Processing Systems, 2021.
[32] T. de Boer, D. P. Kroese, S. Mannor, and R. Y. Rubinstein. tutorial on the cross-entropy method. Annals of Operations Research, 134:19-60 67, 2005.
[33] M. Kelly. An introduction to trajectory optimization: How to do your own direct collocation. SIAM Review, 59(4):849-904 904, 2017.
[34] M. Posa, C. Cantu, and R. Tedrake. A direct method for trajectory optimization of rigid bodies through contact. The International Journal of Robotics Research, 33(1):69-81, 2014.
[35] Z.-Q. Luo, J.S.. Pang, and D. Ralph. Mathematical programs with equilibrium constraints. Cambridge University Press, 1996.
[36] G. Williams, N. Wagener, B. Goldfain, Drews, M. Rehg, B. Boots, and E. A. Theodorou. Information theoretic mpe for model-based reinforcement learning. In 2017 IEEE Interna- tional Conference on Robotics and Automation (ICRA), pages 1714-1721. IEEE, 2017.
[37] A. Venkatraman, R. Capobianco, L.. Pinto, M. Hebert, D. Nardi, and J. A. Bagnell, Improved learning of dynamics models for control. 2016 International Symposium on Experimental Roboties, pages 703-713. Springer. 2017.
[38] T. G. Thuruthel, E. Falotico, F. Renda, and C. Laschi. Learning dynamic models for open loop predictive control of soft robotic manipulators. Bioinspiration & Biomimetics, 12(6): 066003, oct 2017. 111.111088//77-111992000 1088/1748-
URL https:
doi. //tp.ddi..ogg/....///
[39] A. S. Polydoros and L. Nalpantidis. Survey of model-based reinforcement learning: Applica- tions on robotics. Journal of Intelligent & Robotic Systems, 86(2):153-173. 2017.
[40] D. Hafner, T. Lillicrap, Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. International conference on machine learning. PMLR, 2019.
[41] P. Wu, A. Escontrela, D. Hafner, K. Goldberg, and P. Abbeel. Daydreamer: World models for physical robot learning. Conference on Robot Learning, 2022.
[42] M. Macklin, M. Müller, and Chentanez. Xpbd: dusition-based simulation of compliant con- strained dynamics. In Proceedings of the 9th International Conference on Motion Games, 2016.
[43] G. Salhotra, I.-C A. Liu, M. Dominguez-Kuhne, and G. S. Sukhatme. Learning deformable object manipulation from expert demonstrations., IEEE Robotics and Automation Letters, 7(4): 8775-8782, 2022. doi:
[44] A. Nair, A. Gupta, M. Dalal, and S. Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09335 2020.

---

[45] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Rohotics transformer for real-world control at scale. arXiv preprint arXiv:22/2.068177 2022.
[46] X. B. Peng, P. Abbeel, S. Levine, and M. van de Panne. Deepmimic: Example-guided deep inforcement learning of physics- based character skills. ACM Transactions on Graphics (TOG), 2018.
[47] Matas, S. James, and A. J. Davison. Sim-to-real reinforcement learning for deformable object manipulation. In Conference Robot Learning, pages 734-743. PMLR, 2018.
[48] M. Laskin, Srinivas, and Abbeel. CURL:
Contrastive unsupervised representations for re- inforcement learning. H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 10339-5500. PMLR, 13-18 Jul 2020. URL https: ptttt:::/pprcceeiinng .ml..rr.s..........mmm.mm..m.... mmlr. press/ v119 / as k 220 html.
[49] D. Yarats, I. Kostrikov, and R. Fergus. Image augmentation all you need: Regularizing deep reinforcement learning from pixels. In 9th International Conference on Learning Repre- sentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL
https://openrent ew net/ / /t//fforug forum? d- 6- 6s TvGaf.
[50] A. Sanchez-Gonzalez, Godwin, T. Pfaff, R. Ying, Leskovec, and Battaglia. Learning simulate complex physics with graph networks. International conference on machine learning, pages 8459 8468. PMLR, 2020.
[51] X. Lin, Y. Wang. Z. Huang, and D. Held. Learning visible connectivity dynamics for cloth smoothing. In Conference on Robot Learning, pages 256 266..66.11 PMLR, 2022.
[52] Z. Huang, X. Lin, and D. Held. Mesh-based Dynamics with Occlusion Reasoning for Cloth Manipulation. In Proceedings of Robotics: Science and Systems, New York City, NY, USA, June 2022. doi: 111.10.1600/RSS22222.XX2
[53] G. Williams, P. Drews, B. Goldfain, J. M. Rehg, and E. A. Theodorou. Aggressive driving with model predictive path integral control. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 1433-1440. IEEE, 2016.
[54] X. Lin, Y. Wang. J. Olkin, and D. Held. Softgym: Benchmarking deep reinforcement learning for deformable object manipulation. Conference on Robot Learning (CoRL), 2020.
[55] A. Jaegle, S. Borgeaud, J.-B. Alayrac, C. Doersch, C. Ionescu, D. Ding, S. Koppula, D. Zoran, A. Brock, E. Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. In International Conference Learning Representations, 2021.
12

---

Figure 6: Environments used in our experiments, with one end-effector. The end-effectors pickers (white spheres). In CLOTH FOLD (left) the robot has to fold the cloth (orange and pink) along an edge (inspired by the SoftGym [54] two-picker cloth fold task). In DRY CLOTH (middle) the robot has to hang the cloth (orange and pink) on the drying rack (brown plank). In THREE BOXES (right), the robot has rearrange three rigid boxes along line.
A Ablations
We use the DRY CLOTH task for all ablations unless specified; the most challenging of our tasks. We provide detailed answers the following questions in Appendix A. Appendix Fig. illustrates the ablations corresponding to each part of the overall method. (1) How do different methods perform in creating optimized dataset D Student? (2) What is the best architecture to learn the task dynamics?
(3) How good is 'D Student compared to the recorded demonstrations? (4) How well does the downstream LfD method handle different kinds of demonstrations?
(5) How does the use of expert state matching affect the downstream L.D?? (6) How do the baselines perform across related morphologies and environment?
We discovered that the Cross-Entropy Method (CEM) is the most effective optimizer for generating a D Stadent from demonstrations. When combined with CEM, the ID CNN-LSTM architecture produces the best results for trajectory optimization. Our optimized Studeent performs similarly to the pre- rre-programmed D which has access to full state information of the environment. By utilizing our chosen downstream LiD method, we can successfully complete tasks with variety of demonstrations and achieve superior performance compared to both D Student and D Teacher- Expert state matching negatively impacts the performance DMfD. Lastly, we found that GAIfO trained on our D Studens outperforms GAIfO trained on the Deeacher and the difficulty of the environment significantly influences the performance of GAIfO and GPIL.
A.1 Ablate the method for creating optimized dataset Student
We answer the question: how do different methods perform creating optimized dataset D Studentt We ablate the optimizer used create D Stwdens from the demonstrations, labeled ABL1 in Fig. 7, and compare the following methods, given state inputs from D Teacker
Random: A trivial random guesser, that serves as lower benchmark.
SAC: An RL algorithm that tries to reach the goal states of the demonstrations.
Covariant Matrix Adaption Evolution Strategy (CMA-ES): An evolutionary strategy that samples optimization parameters from multi-variate Gaussian, and updates the mean and covariance at each iteration.
Model Predictive Path Integral (MPPI): An information theoretic MPC algorithm that can support learned dynamics and complex cost criteria [36, 53].
Cross-Entropy Method (CEM, ours): A well-known gradient-free optimizer, where we assume a Gaussian distribution for optimization parameters.
13

---

Random
Optimized Demos end-effectors
Actions
end-effectors
ABL 3,4
Learned Spatio-temporal Dynamics Model
Indirect Trajectory Optimization
ABL 2
ABL
Teacher Demos
n end-effectors
Method
ABL 5
Figure 7: Ablations MAIL components.
We did not gradient-based trajectory optimizers since the contact-rich simulation will give rise to discontinuous dynamics and noisy gradients. As shown in Table la, SAC is unable improve upon the random baseline, likely because of the very large state-space of our environment (> 15000 states for > 5000 cloth particles) and error accumulations from the imprecision of learned dynamics model. Trajectory optimizers achieve the highest performance, and we chose CEM as the best optimizer based on the performance of the optimized trajectory.
A.2 Ablate the dynamics model
We answer the question: what is the best architecture to learn the task dynamics? We ablate the learned dynamics model Tg, labeled ABL2 in Fig. 7. The environment state is the state from Deeacher Le., positions of cloth particles. This a structured but large state space since the cloth is discretized into 5000 particles.
Table 1b shows the performance of trajectories achieved by using the dynamics models. We see that CNN-LSTM models work better than models that contain only CNNs, graph networks (GNS [50]), transformers (Perceiver 10 [55]), or LSTMs. We hypothesize that this is the case since we need to capture the spatial structure of cloth and capture temporal element across the whole trajectory since particle velocity is not captured in the state. Further, a 1D CNN works better because the cloth state can be simply represented as a 2D vector (N x which represents the xyz for N particles). This is easier learn with than the 3D state vector fed into 2D CNNs.
GNS performs poorly also due the reasons of error accumulation from large displacements, dis- cussed in Sec. 4.2. Although Perceiver I0 did not perform well CNN-LSTM, it did not affect the downstream performance for the LfD method. We conducted an experiment to compare DMfD performance when it was trained on the D Student obtained from Perceiver IO and CNN-LSTM and found that they had comparable results, shown in Fig. This indicates that MAIL adaptable to different Student and capable of learning from suboptimal demonstrations.
Our learned dynamics model Tr was significantly faster than the simulator: We tested it on simple training run of SAC [5], without parallelization. Our learned dynamics gave 162 fps, about 50x faster than the 3.4 fps with the simulator. However, the dynamics error was not insignificant. We compute the state changes in cloth by considering the cloth particles as a point cloud, and computing distances between point clouds using the chamfer distance. We then executed actions on the cloth for the DRY CLOTH task, comparing the cloth state before action with the model's predicted state and the simulator's true state after the action. Over 100 state transitions, observed cloth movement of 0.67 in the true simulator, and error of 0.17 m between the true and predicted
14

---

y
Simulator
Tw
Simulator
Figure 8: Predictions of the learned spatio-temports dynamics model Tp and the FleX simu- lator. Predictions are made for the same state and action, shown for both cloth tasks. The learned model supports optimization approximately 50.r faster than the simulator, albeit at the cost of accu- racy.
state of the cloth. This accuracy was tolerable for trajectory optimization, qualitatively shown in Fig. 8, where we did not need optimal demonstrations.
A.3 Compare performance of optimized dataset Diptim
We answer the question: how good is D Student compared to the recorded demonstrations? This ablation gauges the performance of the op- timized dataset that we used as the student dataset for LfD, D Student = Doptimm We compare this to other relevant datasets to solve the task, as shown in Table lc. It labeled ABL3 in Fig. 7. The two-picker demonstra- tions D2p Demo are recorded for an agent with two pickers as end-effectors. This is used as the teacher demonstrations in our experiment D Dreacher = D2p Dem. The one-picker demon- strations Demo are recorded for agent with one picker as an end-effector. This is to contrast against the optimized demonstrations in the same morphology, Dipptim The random action trajectories are with a one-picker agent, added a lower performance benchmark. They are the same random trajectories used to train the spatio-temporal dynamics model T.. Naturally, the teacher dataset is the best, as it is trivial to do this task with two pickers. The one-picker dataset has about the same performance as the optimized dataset Dip Optimm both of which are suboptimal. It can be inferred that it is not trivial to manipulate cloth with one hand. This is the kind of task we wish unlock with this work: tasks that easy to do for teachers in morphology but difficult program record demonstrations for in the student's 's morphology. Note that D Optim has been optimized on the fast but inaccurate learned
CW 15TM
PeokkiO
ES-
Noroo Peffrmencee
CLoIE FOLD
Day CLOm
Figure 9: Performance comparison between DMID trained DD PStudent obtained using dif- ferent learned dynamics models: ID CNN-LSTM and Perceiver IO. For each training run, used the best model in each seed's 's training run, and evaluated using 100 rollouts across 5 seeds, dif- ferent from the training seed. Bar height denotes the mean, error bars indicate the standard devia- tion.
15

---

combination of datasets obtained from dynamics model, which is one reason for the reduced performance. This is why the downstream LfD method uses the simulator, as accuracy is very important in the final policy.
Method
25th
14 14
median
75th
Random
0.000
0.003+000088
0.000
0.000
SAC
0.000
0.000+000006
0.000
0.000
CMA-ES
0.104
0.270:10.258
0.286
0.489
MPPI
0.070
0.289.10.264
0.275
0.474
CEM
0.351
0.502-10.242
0.501
0.702
(a) Ablation the method chosen for creating demonstrations.
Method
25 %
+
median
75 %
Perceiver I0
0.305
0.450:10.258
0.486
0.628
GNS
-0.182
0.002:10:223
-0,042
0.149
2D CNN, LSTM
0.157
0.376+t03005
0.382
0.602
No CNN, LSTM
0.327
0.465++0:213
0,463
0.595
ID CNN, No LSTM
0.202
0.407-10.237
0.387
0.587
ID CNN, LSTM (ours)
0.351
0.502-10.242
0.501
0.702
(h) Ablation on the dynamics network architecture.
Dataset
25%
J
median
75th
Random
0.000
0.003-10.088
0.000
0.000
D Demo
0.344
0.484:10.169
0.446
0.641
Do
Demo
0.696
0.744:10.068
0.724
0.785
Dip
Optim
0.351
0.502++0.242
0.501
0.702
(c) Compare the performance of the optimized dataset.
Table 1: Ablation results for MAIL
A.4 Ablate modality of demonstrations
We answer the question: how well does the downstream LID method handle different kinds of demonstrations? This ablates the composition of the student dataset fed into LfD, and is labeled ABL4 in Fig. 7. We compare the following datasets for D Studentt using the notation for datasets explained in See. 3.1:
Demonstrations in one-picker morphology, D.omoo These are non-trivial create and are thus not performant, discussed above. Creating these is increasingly difficult as the task becomes more challenging.
Optimized demos, Diputmm This is optimized from the two-picker teacher demonstrations
(Dreacker = 22g D. which are easy to collect as the task is trivial with two pickers. 50% Dip Demo and 50% Dip A mix of trajectories from the two cases above. This is an example of handling multiple demonstrators with different morphologies.
Fig. 10 illustrates that all three variants achieve similar final performance. This demonstrates that the downstream LID method capable of solving the task with variety of suboptimal demonstrations. This could be from dataset of demonstrations, heterogencous set of teachers.
16

---

0.85
0.50
Performance
0.75
Performance
0.75
0.70
0.05
0:65
0.60
0.1-
0.1-
one derraa
DMDD withor
0.3
Million Steps
Million Steps
Figure 10: Ablation on the modality of demon- strations on LfD performance. Similar perfor- mance shows that MAIL, can learn from a wide variety of demonstrations, or even mixture of them, without loss in performance. See Sec. A.4.
Figure 11: Ablation on the effect of reference state initialization (RSI) and imitation reward (IR) on L.f performance. RSI is not helpful here because our tasks are not dynamic or long horizon as DeepMimic [46]. See Sec. A.S.
An interesting observation here is that by comparing Fig. 10 and Table 1c, that the final policy is better than the suboptimal demonstrations by a considerable margin, and also slightly im- proves upon the performance of the teacher demonstrations. This improvement comes from the LfD method's ability to effectively utilize demonstrations and generalize across task variations. This re- sult, combined with the ablation that we need demonstrations in Sec. 4.2, shows that our downstream LfD method is well adapted work with suboptimal demonstrations solve task.
A.5 Ablate Reference State Initialization in DMID
We answer the question: how does the use of demonstration state matching affect the downstream LfD? An improvement expert states, known as RSI-IR, first proposed in [46]. We justify this improvement in this ablation, labeled ABLS in Fig. 7.
made over the original DMfD algorithm is disable matching with
As shown in Fig. 11, removing RSI and IR has a net positive effect throughout training, and around 10% on the final policy performance. This means that matching expert states exactly via imitation reward does not help, even during the initial stages of training when the policy is randomly initial- ized. We believe this is because RSI helps when there are hard-to-reach intermediate states that the policy cannot reach during the initial stages of training. This is true for dynamic long-horizon tasks, such as karate chops and roundhouse kicks. However, our tasks are quasi-static, and also have a short horizon of 3 for the cloth tasks. other words, removing this technique allows the policy to freely explore the state space while the demonstrations can still guide the RL policy learning via the ad udvantage-weigee loss from DMID.
A.6 Ablate the effect of cross-morphology LfD baselines
We answer the question: how do established LfD baselines perform across morphologies? This ablation studies the effect of cross-morphology in the demonstrations, where compare the per- formance of GAIfO, when provided demonstrations from the teacher dataset D Teacher and (subop- timal) student dataset D Student: for the DRY CLOTH task.
As we can see Table 2, there is 36% performance improvement when using D Sttudent instead of Deeacher- The primary difference that the agent sees during training is the richness of demonstration states, the demonstration actions has only picker, any demonstration for the task (DRY CLOTH) includes multiple intermediate states of the cloth in various conditions of being partially hung for drying. By contrast, the teacher
available learn from. Since the student morphology
17

---

requires fewer plcce steps to complete the task, and thus there are fewer intermediate states in the demonstrations.
A.7 Ablate the effect of environment difficulty on LfD baselines
We answer the question: how established LfD baselines perform across environments? Given the subpar performance of the LfD baselines GAIfO and GPIL on our SOTA environments, ablated the effect of environment difficulty. We took the easy cloth environment (CLOTI FOLD) and used easier variant of it, CLOTH FOLD DIAGONAL PINNED [43]. In this variant, the agent has to fold cloth along diagonal, which can be done by manipulating only one corner of the cloth. Moreover, one comner of the cloth is pinned to prevent sliding, making it easier to perform. We used state-based observations and -ispaaeeiplacemeett action space, where the agent outputs incremental picker displacements instead of pick-and-plas locations. We can see in Table that the same basclines are able to perform significantly better in this environment. Hence, we believe manipulating with horg-horizon pick-place actions, with image observation, makes it challenging for the baselines to perform cloth manipulation tasks described in Sec. 4.1 and Appendix B.
Method
25%
1 +
median 75%h
Dreacher -0.198
-0.055+00.183
-0.043
0.078
Dstudent
0.199
0.363.10.245
0.409
0.528
Table Ablation of GAIfO on the effect cross-morphology. We compare normalized perfor- mance, measured at the end of the task.
Method
25%
μA +
median 75th
GPIL
0.356
0,427=1.162
0.487
0,553
GAHO
0.115
0.374:00267
0.471
0.592
Table 3: Measuring performance the easy cloth task, CLOTH FOLD DIAGONAL. PINNED, We compare the normalized performance, measured at the end of the task.
B
Tasks
Here we give more details about the tasks, including the performance functions, teacher dataset, and sample images. Fig. 6 shows images all of simulation environments used for SOTA comparisons and generalizability, with end-effector. In each environment, the end-effectors are pickers (white spheres). In cloth-based environments, the cloth is discretized into 80x80 grid of particles, giving total of 6400 particles.
1. CLOTH FOLD: Fold square cloth in half, along specified line. The performance metric is the distance of the cloth particles left of the folding line, those the right of the folding line. A fully folded cloth should have these two halves virtually overlap. Teacher demonstrations are from an agent with two pickers (i.e., Deeache = 2p D we solve the task on student agent with one picker. Task variations are in cloth rotation.
2. DRY CLOTH: Pick up a square cloth from the ground and hang it on a plank to dry, variant [47]. The performance metric the number cloth particles (in simulation) on either side of the plank and above the ground. Teacher demonstrations are from an agent with pickers (i.e., Dreacher D solve the task on student agent with one picker. Task variations are in cloth rotations and translations with respect to the plank.
18

---

3. THREE BOXES: A simple environment with three boxes along a line that need to be rear- ranged to designated goal locations. Teacher demonstrations are from an agent with three pickers (i.e. Teacher = 3p Demo): we solve the task on student agents with one picker and two pickers. Performance measured by the distance of each object from its goal location. This task used illustrate the generalizability of MAIL with various n-to-m end-effector transfers, and is not used in the SOTA comparisons.
C Hyperparameter choices for MAIL
In this section, Table 4 shows the hyperparameters chosen for training the forward dynamics model Tø Table 5 shows the details of CEM hyperparameter choices. Table 6 shows the hyperparameters for our chosen LfD method (DMfD).
Parameter
Description
4 layers, 32 channels, 3x3 kernel, leaky ReLU activation. stride = for the first layer, stride for subsequent layers One layer
CNN
LSTM
Hidden size = 32
Learning rate =le-5
Other Parameters
Batch size 128
Table 4:
for training the forward dynamics model.
Planning Horizon
Number of optimization iterations
Number of env interactions
1
2
21,000
2
2
2
15,000
3
21,000
4
2
31,000
5
2
2
34,000
10
21,000
7
21,000
8
2
15,000
9
32,000
10
2
21,000
11
10
21,000
12
21,000
13
10
21,000
Table 5: CEM hyper-parametert tested for tuning the trajectory optimization. We conducted ten rollouts for each parameter set and used the set with the highest average normalized performance on the teacher demonstrations. Population size is determined by the number of environment inter- actions. The number of clites for each CEM iteration 10% population size.
D Performance metrics for real-world cloth experiments
In this section, we explain the metries for measuring performance of the cloth, to explain the sim2real results discussed in Sec. 4.2.1
For CLOTH FOLD task, we measure performance at time by the number of pixels of the top color pistoopt and bottom color pio of the flattened cloth, compared to the maximum number of pixels, mixmaa (Fig. 12).
For DRY CLOTH task, is challenging measure pixels the sides and top of the plank. Moreover, could be double counting pixels if they are visible in both side and top views. Hence, we measure
19

---

Parameter
Description
Fully connected network (FCN)
State encoding
2 hidden layers of 1024, ReLU activation
32x32 RGB input, with random crops.
Image encoding
CNN: layers, 32 channels, stride 3x3 kernel, leaky ReLU activation
FCN:
layer of 1024 neurons, tanh activation
Fully connected network
Actor
2 hidden layers of 1024, leaky ReLU activation
Fully connected network
Critic
2 hidden layers of 1024, leaky ReLU activation
Discount factor: 7 = 0.9
Entropy loss weight: wg 0.1
Entropy regularizer coefficient: = 0.5
Other parameters
Batch size 256
Replay buffer size 600.000
RSI-IR probability = (disabled)
Table Hyper-parameters used the LfD method (DMfD).
1.0
Top metric p(top)
Bottom metric p(bot)
0.8
Performance
0.6
0.4
0.2
0.0
0.00
0.25
0.50
0.75
1.00
Fraction of pixels frop, fbot
measure the fraction pixels visible to the maximum number of pixels visible fiop = piztop,t/ pix and foot pirbot,t/ ppirmaa- Performance for the top of the cloth should be 1 when it is not visible, p(top) 1 frop- Performance for the bottom of the cloth should be 1 when it is exactly half- folded top of the top side, p(bol) min [2 (1 Shot) 2bor). Final performance an average of both metrics, p(sx) = p(lop) + p(bootoom) Note that the cloth is flattened at the start, thus pixmaax pirrop.0-
Figure 12: Performance function for CLOTH FOLD the real robot. At time
20

---

the cloth to determine whether the length of the cloth rop of the plank is equal to or greater than the side of the square cloth. We call this the spread metric.
The policies achieve 80% performance, which is about the average performance of our method in simulation, for both tasks. However, since these performance metries are different in the simulation and real world, cannot quantify the sim2real gap through these numbers.
E Collected dataset of teacher demonstrations
We have 100 demonstrations provided by the teacher, mentioned Sec. 3.4. The diversity of the task comes from the initial conditions for these demonstrations, which are sampled from the task distribution Dd ~ V. This variability in the initial state adds diversity to the dataset. The quality and performance of these teacher demonstrations were briefly discussed in the ablations (Sec. A.4).
All demonstrations come from a scripted policy. For ClothFold, the teacher has two end-effectors and picks two comners of the cloth to move them towards the other two corners. For DryCloth, the teacher has two end-effectors and picks two corners of the cloth to move them to the other side of the rack. They maintain the same distance between each other during the move to ensure the cloth is spread out when hangs on the rack. For Toxee the teacher has three end-effectors. It picks up all the boxes simultancously and places them in their respective goals.
Random actions dataset used for training the dynamics model
We trained the dynamics model on random actions from various states, to cover the state-action distributions our tasks would operate under.
For CLOTH FOLD, random action policy is to pick random cloth particle and move the particle to random goal location within the action space. For DRY CLOTH, the random action policy is to pick random cloth particle, and move it to a random goal location around the drying rack, to learn cloth interactions around the rack. For completeness, we also trained a forward dynamics model for the THREE BOXES task. Here, the random action policy to pick the boxes order and sample a random place location within the action space.
Each task's episode horizon is 3. Our actions are pick-and-place actions, and the action space is in the full range of visibility of the camera. For DRY CLOTH, this limit is -0.5, O, 0.5] to [0.5. 0.7, 0.5). For CLOTH FOLD it is [-0.9, 0, 0.9] to [0.9, 0.7, 0.9]. For THREE BOXES it is -0 0.1 to 1.35.
21


Reviews:
[{"review_id": "_CC1c17IOB2", "reviewer": "Reviewer_UU6D", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper studies learning from demonstrations that have substantially different morphologies than the current robot, hence the use of the term \"cross-morphology\". The method is named \"Morphological Adaptation in Imitation Learning (MAIL)\" and it tries to tackle policy learning for a robot with $m$ end-effectors from demonstrators with $n$ end-effectors, where in general $n \\ne m$. Since the end-effector count is not necessarily equal, the key idea of MAIL is not to match morphologies but to instead match the object states instead. The paper tests on robotic cloth manipulation applications including simulation and in the real world, along with a simulated block arrangement task. They show that their method is able to outperform numerous competing baseline methods.\n", "summary_of_recommendation": "This method and paper has some promise. There are a few concerns I have as noted in the review about how much this will generalize to $m$ and $n$ end-effectors as well as some confusing aspects of the text. \n\nUpdate: based on the author response where they said they would modify the writing of the paper about the \"any n-to-m transfer\" part, I'm OK with a weak accept.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "I don't have specific questions other than those implied by the review above. Please correct me if I happen to be mistaken in my understandings.\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths: \n\n1. Learning from cross-morphology demonstrations is an important problem especially when we consider humans providing demonstrations to robots, due to completely different morphologies.\n\n2. While the paper only tests on two variants of cloth manipulation (as well as a less-interesting simulated block arrangement task), it benchmarks against a lot of other methods, so I appreciate that rigorous experimentation. There are multiple variants of SAC.\n\n3. Cloth manipulation is itself challenging and I appreciate their efforts in doing so.\n\nAreas to potentially improve:\n\nI think I get the rough idea of the method but there are a lot of confusing aspects to the paper which hinder my understanding. Here are my confusions:\n\n- While the method, MAIL, is advertised as an imitation learning method, it is really a combination of imitation learning and reinforcement learning, with the RL part happening later. This could make it harder to apply for robotics tasks that don't have a suitable simulator. Maybe the RL aspect should be emphasized earlier just so that the reader is aware that RL will be needed.\n\n- Similarly in terms of clarification, MAIL specifically focuses on *end-effector count* morphology, such as using 1 picker or 2 picker. This is somewhat different from what I had thought of originally which could have referred to any human demonstration versus robot demonstration (as humans and robots have different morphologies).\n\n- There are two places in the text that suggest cloth manipulation is very slow to simulate, in lines 62-63 (\"These can significantly slow down simulation…\") and in lines 147-152 (\"Robotic simulators have come a long way in advancing fidelity and speed, but [...] making optimization intractable for challenging simulations\"). It took me a while to understand what the paper was saying, since I had assumed their method would try to *avoid* any cloth simulation due to \"intractability\" but the paper quickly says that it will use NVIDIA FleX simulation anyway! After reading carefully, I think the point is that MAIL still uses NVIDIA FleX, it just doesn't use it for \"forward dynamics\", instead it uses a learned dynamics model which imitates the simulator but is faster to query than the simulator itself. I think that's the reason (please correct me if I am wrong) but I fear the writing is confusing in that regard.\n\n- As a side comment, I have used NVIDIA's cloth simulation in the past and I found the simulation to be quite fast to run, hence I'm surprised that the paper reports a 50X speed-up (162 fps vs 3.4 fps) with the learned dynamics vs the underlying simulator. Since the paper uses CEM for optimization, isn't it possible to parallelize the underlying FleX simulation to get faster data (even if each individual environment runs at the same speed)?\n\n- Lines 152-153: the paper says it uses a CNN-LSTM based forward dynamics model. But then why are the inputs represented as N particle positions? Normally, a CNN would take in an image as input, right? Taking in particles would imply using a graph neural network or a point cloud neural network, where I might interpret a \"particle\" as a graph node or a single point, respectively. Incidentally, why use CNNs and LSTMs at all if one can use Transformers?\n\n- Sections 3.3 and 3.4: I was getting confused about why the learned dynamics model was used in 3.3, yet the underlying simulator was used in 3.4. Two comments: first, maybe it would help to clearly identify when the learned dynamics versus the underlying simulator is used in the different stages of MAIL. Second, since the underlying simulator is used in 3.4 anyway, that seems to suggest the underlying cloth simulation is fast enough? Just another thought: maybe it helps to identify the rough computational resources needed for both 3.3 and 3.4 to understand which of those two stages is the time bottleneck.\n\nOther concerns have to do with some possible over-claiming:\n\n- Line 72 says that \"MAIL can handle general instances of $n$-to-$m$ end-effector transfer. However this is not what is being shown. Instead, as line 74 says, the specific example MAIL shows is handling 3-to-2, 3-to-1, and 2-to-1 end-effector transfer. Saying \"general instances\" implies that we can have $n$ and $m$ be arbitrary integers. Thus I would suggest changing line 72 accordingly.\n\n- Line 318 to 319: similarly, the paper says \"Thus, MAIL is capable of general $n$-to-$m$ end-effector transfer. This is not what was shown. Also just before this, line 316, I would change \"Thus, MAIL can solve a task using…\" to \"Thus, MAIL can solve this particular task using…\" because this is focusing on the particular task of block rearrangement.\n\n- Line 368 (in the conclusion), similarly, the paper says \"We further showed LfD generalizability to any transfer from $n$-to-$m$ end-effectors.\" I suggest rewriting the sentence to say \"3-to-2, 3-to-1, and 2-to-1\" because that is what was actually shown in experiments, and it should condition on the specific tasks that were shown.\n\nFinally, one comment has to do with related work. I would recommend adding more material about cross-morphology learning. For example, cross-embodied IRL https://x-irl.github.io/ and AVID https://arxiv.org/abs/1912.04443 which tries to translate demonstrations from humans to robots. These are just two examples I am aware of; there are likely many more especially if we count imitation from dexterous manipulators (e.g,. Human hand vs Allegro hand). I would probably de-emphasize the generic \"Imitation from Observation\" portion unless those specifically deal with the challenges of translating from human demonstrations to robot demonstrations.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "LDx1Nlrxt-", "reviewer": "Reviewer_nE7X", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces a framework called Morphological Adaptation in Imitation Learning (MAIL), enabling learning from demonstrations across robots with different morphologies. It addresses the challenge of training a robot with a different number of end-effectors compared to the teacher robot. Using trajectory optimization, the framework converts state-based demonstrations into suboptimal trajectories in the student's morphology. These trajectories are then used by a Learning from Demonstrations (LfD) method to improve the student's performance by interacting with the environment.\n\nThe paper's main contributions are the proposal of the MAIL framework that allows training a robot with demonstrations that has a morphological mismatch between end-effectors. The paper demonstrates MAIL's effectiveness on manipulation tasks involving rigid and deformable objects, including 3D cloth manipulation with obstacles. The policy learned using MAIL also transfers successfully to a real robot.", "summary_of_recommendation": "I think the paper demonstrates a novel framework and presents valuable contributions in the field of morphological adaptation in imitation learning.\n\n-----------------------------\n\n**Post rebuttal**\n\nThe author had addressed some of my concerns on the contribution and the lack of details in the setup. I think this work does have value in domain adaptation for learning from demonstration. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "1. Some technical details related to trajectory optimization and adapting LfD methods could be further clarified. In particular, the sole purpose of the learned dynamics model seems to be only for faster simulation speed for generating the student dataset (and presumably, this would be an offline process). The same model is not used for the actual LfD (line 199). What is the effect of using the learned dynamic as a proxy for generating the student dataset?\n2. Moreover, for the metric in Figure 3, are they comparing against the student dataset? If so, wouldn't this be affected by the accuracy of the trajectory optimizer?\n3. The typical optimizers for robotics seem to have not been considered. E.g., STOMP, CHOMP, TrajOpt, etc. Some of which are also gradient-free.\n\n\nMinor comments:\n- The notation style in Figure 2 mismatches with the rest of the paper.\n- It would be better if the figure could be self-contained. Many of the minimization objectives presented in the figure are unclear (due to undefined notations) unless you jump to their respective sections.\n- Are there performance metrics for the Three-Box task?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper demonstrates a high level of quality in terms of its methodology, experimental setup, and results. The authors propose a novel framework, MAIL, and provide a detailed explanation of its components. The experiments are well-designed and thorough, showcasing the effectiveness of the framework on challenging cloth manipulation tasks . The use of both simulated and real-world scenarios adds to the robustness of the study.\n\nClarity: The paper is generally well-written and effectively conveys the main ideas and contributions of the work. The authors provide clear explanations of the framework and its components, making it accessible to readers. However, some sections could benefit from further clarification, especially regarding specific technical details related to trajectory optimization and the adaptation of LfD methods.\n\nOriginality: The paper introduces a novel framework, MAIL, which addresses the problem of learning from demonstrations across robots with different morphologies. While the concept of LfD is not new, the specific focus on morphological adaptation is a novel contribution. The approach of converting state-based demonstrations into suboptimal trajectories and leveraging trajectory optimization for speed and efficiency is also original. The combination of these elements distinguishes the paper from previous works.\n\nSignificance: The work presented in the paper has significant implications for the field of imitation learning and robotics. The ability to train robots with different morphologies using demonstrations from other robots expands the possibilities for knowledge transfer and robot learning. The demonstrated improvements over baselines and successful deployment on a real robot highlight the practical relevance of the framework. The generalizability of the approach to different tasks and end-effector transfers further enhances its significance.\n\nStrengths:\n1. Introduction of the novel framework MAIL, addressing the problem of morphological adaptation in LfD.\n2. Thorough experimental evaluation showcasing the effectiveness of MAIL on challenging manipulation tasks.\n3. Successful deployment of the learned policy on a real robot, demonstrating practical applicability.\n4. Generalizability of the framework to various instances of end-effector transfer, illustrating its versatility.\n\nWeaknesses:\n1. Limited Discussion on Limitations: The paper does not extensively discuss the limitations of the proposed framework. Addressing potential challenges, constraints, or scenarios where the framework might not perform well would provide a more balanced perspective on its applicability. In line 317, where n<=m, the author provided a method to tackle mismatch when the teacher has less action space than the robot. However, such an approach would limit the capability of the robot and seems to be a wasteful approach.\n2. Incomplete Technical Explanations: Some technical aspects, such as trajectory optimization and the adaptation of LfD methods, could be further clarified. Improved explanations and additional details would help readers better understand the proposed framework and its underlying mechanisms (as creating the optimized student dataset seems to be a key point in this paper).\n3. Limited Exploration of Alternative Optimization Techniques: The paper primarily focuses on trajectory optimization as the means to convert demonstrations. Exploring and discussing alternative optimization techniques or considering the limitations of the chosen approach would enhance the robustness of the framework.\n4. Lack of Discussion on Dataset Variability: The paper does not discuss the variability or diversity of the datasets used for training and evaluation. Exploring the impact of dataset characteristics, such as the number of demonstrations, their quality, or the diversity of tasks, would provide deeper insights into the framework's performance.\n", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "IQtuPJFvUc", "reviewer": "Reviewer_ExJj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Authors propose a novel learning from demonstration (LfD) framework MAIL which allows for teacher and student morphologies to vary greatly. The particular discrepency they focus on are when the number of end effectors differ. Their method consists of learning a dynamics model which is used by a trajectory optimiser to convert demonstrations into the student's morphology. Finally, this trajectory is used by an existing LfD method which learns a policy in state and image space, intended to generalise to arbitrary initial environment states. They demonstrate their method in a wide range of experimental scenarios, both in simulation and hardware, and show their method outperforms baseline methods and is able to generalise to different morphologies.", "summary_of_recommendation": "The main strength of this paper is the experimental section. However, overall the paper lacks originality combined with vague method descriptions.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "For the 4.3 experiment I believe you are training separate policies for each student morphology as opposed to one policy which generalises to different morphologies. If so, can you please make this clearer in the text?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Quality/ Clarity\n--------------------\nPaper is overall well written and presented. However, some sections are too reliant on appendices and cited work. For example, a short description of DMfD would help to make the paper more self-contained. Furthermore, the abalation study is not really useful without the results. Either incorporate results into the section or move entirely into appendix.\n\nOriginality\n---------------\nThe proposed method is mainly a combination of existing methods. Utilising full state information for task transfer and then training the policy on images is an interesting idea but may be restrictive for certain problem settings.\n\nSignificance\n-----------------\nThe proposed framework is shown to work well and is mostly practical enough to be utilised. Being able to demonstrate a task naturally and transfer to a robot with a significantly different embodyment has real use-case scenarios.\n\nStrengths\n----------------\n- Method works well and generalises to varying inital conditions\n- Thorough experiments and shown to transfer to real world hardware\n- Video is nicely put together and helps in understanding the paper\n\nWeaknesses\n-------------------\n- Question around novelity since this is mainly a trivial combination of existing work\n- Applicability to certain problem settings may be limited due to requiring full state information\n- Method explanations are lacking and could be more self contained rather than simply citing work being used\n- Some sections rely too much on content in appendix\n\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "6vcfxa8Ey8N", "reviewer": "Reviewer_igC1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents MAIL, a new imitation learning algorithm for learning across large differences in teacher and student action spaces. For example, the teacher could demonstrate with two hands while the student robot must use a single manipulator. The key idea in the approach is to perform trajectory optimization in the student's action space such that the distance between the final state in the student trajectory and that of the teacher's trajectory is minimized. To make this process tractable when forward simulation is computationally expensive, the authors propose training a forward dynamics model that can be queried instead. The student dataset created through this process can then be input to an off-the-shelf IL or RLfD (reinforcement learning from demonstration) algorithm that produces a proficient student policy. The authors evaluate their method on cloth manipulation and box rearrangement in simulation and demonstrate transfer to real hardware.", "summary_of_recommendation": "**Post-Rebuttal Update**: I am keeping my Weak Accept score. The authors have addressed most of my concerns, but I am still not convinced that the method is particularly novel. I still recommend acceptance as the paper has other strengths.\n\nI recommend acceptance contingent on making the paper clearer in the ways I have noted in the review. This should be very doable in the rebuttal period as I haven't explicitly asked for any additional experiments. However, if the other reviewers disagree, especially if they have found highly related work I am not aware of, I may reconsider.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "See weaknesses in the Review section.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose an intuitive recipe for addressing a challenging and important problem in robotics: cross-morphology imitation learning. Progress in this area can unlock significant advances, such as robot learning from videos of humans as well as reusable expert data across a range of robot morphologies. The proposed method is technically sound and achieves strong improvement over baselines in the experiments. Cloth manipulation is a difficult task, and the video attachment demonstrates the proficiency of the single-picker student in accomplishing the task demonstrated by the dual-picker teacher. \n\nThe paper has some weaknesses as well. First, the method is simple: essentially, it simply trains a forward dynamics model then performs the cross-entropy method, which is a well-known formula (e.g., visual foresight [https://arxiv.org/abs/1812.00568]). While the dynamics model is motivated by the use of the expensive cloth simulator, even this component is not required if fast forward simulation is available. Its application to cross-morphology IL is novel as far as I know, but given the large body of work in imitation from observation (IfO) that I am not fully familiar with, I would not be surprised if another reviewer is aware of a similar work that has tried this approach. Second, evaluation is significant but a bit limited. The rearrangement task is very simple as the authors note, and the cloth manipulation involves only two softgym tasks. Also, the real-world evaluation is limited to demonstrating sim2real policy transfer rather than learning from real data. Given that there is no simulator of the real world, it would be interesting if a visual dynamics model learned on real data could be used in the proposed method. Third, the clarity of the paper would benefit from several clarifications and added details, some of which I list below: \n\n- How is the dataset of random actions for training the dynamics model generated? What is the random action policy, the range of the action deltas, the episode horizon, etc? This would be good to add to the appendix as adequate coverage of the state space is essential for the dynamics model to be useful in MAIL.\n- How many particle positions N are used in the state representation?\n- Section 3.3 says \"we use them with the simulator to obtain the optimized trajectories\" -- does D_student consist of states from the dynamics model or the simulator?\n- How are demonstrations provided? Is it a scripted policy? What are the details of the scripted policy?\n- Figure 3 shows mean and error bars across seeds - are these training seeds or evaluation seeds? If the latter, is there only a single training seed per algorithm?\n- The sim-to-real gap appears to be extremely large for the Three Boxes task -- how does an image-based policy trained in sim transfer zero-shot to the real world in this case? Is there a state estimation pipeline in real?\n- The authors note an emergent behavior of straightening the cloth out on the plank. Is this behavior captured in D_student or generated by the LfD algorithm?\n- The authors note GNS performs poorly because it is trained on small action deltas. Why not just train it on the larger action deltas for a more fair comparison?", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduced a cross-morphology imitation learning method, which aims to learn from demonstrations where the teacher robot's morphology is significantly different from the student's, e.g., different numbers of end-effectors. The proposed approach leveraged a learned forward dynamics model to perform the cross-entropy method such that the distance between the final state in the student's trajectory and that of the teacher's trajectory is minimized. Evaluations were conducted on cloth manipulation and box rearrangement tasks in simulation, and sim2real results have been shown on a physical robot arm.

This paper received positive ratings from four reviewers, who gave Weak Accept after the rebuttal. However, there was a lack of enthusiastic support. Some lingering issues were brought up by the reviewers, including the limited novelty of the proposed method and the experiments on toy problems, which are also acknowledged by the authors to some extent. Toward the end of the discussion period, Reviewer ExJj expressed that "Inclined to reject this paper since authors have been unable to convince me that their approach isn't just a trivial combination of existing methods." Reviewer igC1 "will keep it as a weak accept as I'm not convinced the method is very novel."

The AC read the paper after going through the reviews, rebuttal, and post-rebuttal discussions in detail. While there is ample room to improve this work, it has offered a reasonable solution to a well-motivated problem. To encourage this line of work, the AC would recommend this paper to be accepted, but with a low confidence, and leave it to the PCs for the final decision.
Decision: Accept (Poster)