HERD: Continuous Human-to-Robot Evolution for Learning from Human Demonstration
Authors: Xingyu Liu
Deepak Pathak
Kris M. Kitani
Venue: CoRL 2022 Poster (2022)
Keywords: learning from human demonstration
imitation learning
curriculum learning
transfer learning
robotic manipulation

Abstract: The ability to learn from human demonstration endows robots with the ability to automate various tasks. However, directly learning from human demonstration is challenging since the structure of the human hand can be very different from the desired robot gripper. In this work, we show that manipulation skills can be transferred from a human to a robot through the use of micro-evolutionary reinforcement learning, where a five-finger human dexterous hand robot gradually evolves into a commercial robot, while repeated interacting in a physics simulator to continuously update the policy that is first learned from human demonstration. To deal with the high dimensions of robot parameters, we propose an algorithm for multi-dimensional evolution path searching that allows joint optimization of both the robot evolution path and the policy. Through experiments on human object manipulation datasets, we show that our framework can efficiently transfer the expert human agent policy trained from human demonstrations in diverse modalities to target commercial robots.

HERD: Continuous Human-to-Robot Evolution for Learning from Human Demonstration
Xingyu Liu
Deepak Pathak Carnegie Mellon University
Kris M. Kitani
OR
Human Demonstrations in Glove Sensor Signals OR Visual Observations
(2) Policy Embodied Human
(3) Transfer Policy via Intermediate Robots from Continuous Hand-to-robot Evolution
Pollcy Target Gripper
Figure 1: HERD framework: (1) Given human demonstrations in sensor glove signals [1] or visual data [2], (2) expert policies on an embodied human hand, i.e. five-finger dexterous robot, can be trained. (3) We then design continuous robot evolution to connect the five- lie-finger dexterous robot -ro-inggr-rrrpperr and target commercial two-finger gripper robot. smooth curriculum of policy optimization on the intermediate robots that gradually evolve towards the target robot allows (4) the expert policies be transferred the target robot even in challenging sparse-reward reward tasks.
Abstract: The ability to learn from human demonstration endows robots with the ability to automate various tasks. However, directly learning from human demon- stration challenging since the structure of the human hand can be very different from the desired robot gripper. In this work, we show that manipulation skills can be transferred from human to a robot through the of micro-evolutionarr reinforcement learning, where a five-finger human dexterous hand robot gradually evolves into a commercial robot, while repeated interacting in physics simulator to continuously update the policy that is first learned from human demonstration. To deal with the high dimensions of robot parameters, we propose algorithm for multi- ditiniiimensioall evolution path searching that allows joint optimization of both the robot evolution path and the policy. Through experiments manipulation datasets, we show that our framework can efficiently transfer the ex- pert human agent policy trained from human demonstrations in diverse modalities to target commercial robots.
human object
Introduction
Learning from human demonstrations [3] is promising direction to enable robots to perform di- verse manipulation capabilities. Recent large-scale visual datasets of human activity [2, 4, 5, 7] have highlighted the need for enabling robotic agents to learn from human activities to perform di- verse tasks in real-world environments. Existing paradigms for learning from human demonstration
6th Conference on Robot Learning (CoRL 2022), Auckland, New Zealand.

---

generally follow the pipeline of demonstration conversion, i.c. (1) convert demonstrated human states to robot states and then (2) train the robot policy. Though step (2) can be solved by existing imitation learning approaches [8. 9, 10. 11, step (1) is extremely difficult. Efforts to address step (1) include human-robot observation matching such as [11], and converting human demonstrations to robots by human-rowe robot interaction [12, 13] or couaa--n--the--e these solutions are highly situational and task-specific and require significant human intervention for each individual task, therefore are not scalable and cannot automatically generalize to new tasks.
teleoperation [14, 15]. However,
We propose new paradigm for learning from human demonstration - policy transformation. i.e. (I) train policy on embodied human hand dexterous robot and then (2) transfer the policy from dexterous hand robot to the target robot. Many existing methods have been shown successful in learning control policy for dexterous hand robots from human demonstration [1, 16] and can easily solve step (1) of our pipeline. A general and automatic solution of step (2) would render scalable solution of learning from human demonstration. The key challenge to step (2) is the huge dynamics discrepancy between the dexterous hand robot and the target robot. The discrepancy stems from the mismatch robot morphology and kinematics. Unfortunately, statistical matching imitation learning approaches such as [8, 9, 10, 17] assume the teacher and student robotic agents share the same or similar transition dynamics, therefore are ineffective in solving the problem.
In this paper, we present a framework named HERD (short for Human Evolution to Robot for Learn- ing from Demonstration) with new perspective on solving the problem of learning from human demonstration - continuous robot evolution. Following the philosophy introduced in the seminal work of REvolveR [18], the core idea of our framework is define a continuous interpolation be- tween the dexterous hand robot and a target robot that have different morphology. Then an expert dexterous hand robot policy learned with methods such as [1, 16] can be transferred to the target robot through training on a sequence of intermediate robots that gradually evolve into the target robot. The policy is continuously updated through repeated interaction in a physics engine that contains the intermediate robots, as illustrated in Figure 1.
To instantiate the idea, we develop two solutions for the evolution from five-finger dexterous robot to two target commercial robots respectively: a Sawyer robot with a two-finger Rethink gripper, and a Jaco robot with a three-finger Jaco gripper. The solutions precisely match the dynamics of target commercial grippers. To deal with the high dimensions of robot parameters, we propose an algorithm for automatically searching evolution paths in high-dimensional robot parameter space that allows joint optimization of both robot evolution and the policy, based on online estimation of reward gradient with respect to robot evolution.
We conduct experiments on transferring human expert demonstrations of manipulation tasks in di- verse modalities to the target commercial robots. We show that HERD can successfully transfer the expert dexterous hand robot policy trained from Hand Manipulation Suite sensor glove demonstra- tions [1] to the target robots even with sparse rewards. On Dex dataset [2], we demonstrate that our approach can also transfer the expert policy trained with visual human demonstrations to the target robots. In addition, we show that the proposed multi-dimensional robot evolution path search algorithm can significantly improve the policy transferring efficiency.
2 Preliminaries
MDP Preliminary We consider a continuous control problem formulated as Markov Decision Process (MDP). is defined by a tuple (S, A, 7), where S C IR. is the state space, A C R is the action space. T S x A - S is the transition function, R S x A - IR is the reward function, and 7 [0, 1] is the discount factor. A policy S A maps state to action where T (a)s) is the probability of choosing action a at state s. Suppose M is the set of all MDPs and pT.M = 7' (t) is the episode discounted reward with policy æ on MDP M E M. The optimal policy Tÿ on MDP M the one that maximizes the expected value of pr.M.
REvolveR [18] Preliminary Liu et al. [18] proposed a technique named REvolveR for transferring policies from source robot target robot. The core idea is define an evolutionary sequence of intermediate robots that connects the source to the target robot. Given source and target robots represented by two MDPs Ms, MM E M respectively. REvolveR defines continuous function E : [0, 1] + M where (() = Ms, E(1 = Mr. Then an expert policy x T on the source robot (()) is optimized by sequentially interacting with each intermediate robot in the sequence
2

---

1
Figure 2: Continuous hand-to-robot evolution for Sawyer robot. We show one evolution path from human dexterous hand (a 0) to commercial Sawyer robot with two-finger Rethink gripper (a 1). From left to right, we show intermediate robots along the path. Zoom for better view.
F((1), F((a2),.... ... F(ox) where < 01 02 <.. = 1, until the policy is able act (near) optimally on each intermediate robot and eventually transfer the policy to target robot E(1). For all |05+1 agl is small enough that each policy fine-tuning step is much easier task. Our method is built upon the philosophy of REvolveR to use robot evolution for policy transfer.
Notation We use bold letters to denote the vector variables. Specially, 0 and 1 are the all-zero and all-one vectors with proper dimensions respectively. 5D)1(() for E R ||>||| {} is
(D !)-shhere (i.e. sphere in IRD space) with radius & R= @ is element-wise product.
3 Method
3.1 Human Demonstration to Dexterous Robot Expert Policy
Recent works [1, 16] have shown success in learning five-finger dexterous robot policy from human hand demonstrations in diverse modalities, such sensor glove signals [1] and visual data [16]. which made the first step of our proposed paradigm possible. Inspired by this research progress, we adopt similar approaches train expert policies on a dexterous robot illustrated in Figure 1(1)(2).
Sensor Glove Demonstration When recorded in sensor glove signals, human demonstrations are directly converted to states and actions in simulator [19] from which a policy is trained with algorithms such as DAPG [1]. We use the same procedure and obtain the expert policy as [1].
Visual Demonstration We develop toolchain for training dexterous policies from human visual demonstrations of manipulation. Our toolchain includes human and object pose estimation with visual perception followed by pose fitting of the dexterous robot via inverse kinematics (IK). We then train the policy by setting the episodes' initial state be the state of the grasping frame, including poses and joint velocity. During training, the initial state gradually moves backward in time until the desired starting state. For more details, please refer the supplementary.
3.2 Continuous Dexterous Hand to Target Commercial Robots Evolution
A core hypothesis in our proposed paradigm of learning from human demonstration is the feasibil- ity of interpolation between a numan-hand-Bae five-finger dexterous robot and target commercial robot. In this paper, we present solutions for cesterous-to-commerria target commercial robots to show this, Since both the dexterous hand robot and the target robots are complicated in structure, how design the evolution from the former the latter not straight- forward and has a large design space. The design choice can significantly affect the policy transfer performance. Following [18], we first match the robot morphology between the dexterous and the target robot. With the same morphology, we then design a shared high-dimensional robot parameter space that allows defining an intermediate robot by simply setting these parameters. The evolution includes both the dexterous hand and arm.
interpolation for two
Dexterous Hand to Commercial Gripper Evolution To match the number of fingers, keep the thumb and gradually shrink the redundant fingers to be zero-size and zero-mass so that they eventually disappear. We also gradually change the range of the finger joints to match the desired
3

---

joint configuration. In addition, we add new position servo joints to the remaining fingers. The position servo joints are initially frozen and evolve to have the same range as the target fingers.
Dexterous Arm to Commercial Robot Arm Evolution A major challenge of arm evolution is the different ways of arm mounting. In simulation, the dexterous robot elbow joint is usually modeled as a virtual free joint that can move freely in the 3D space, while the commercial robot arms are usually mounted on fixed bases. Another challenge is the huge difference in the numbers of arm joints and joint physical parameters. Our solution is to attach the free joint of the dexterous arm to the end effector of the target robot and treat the entire dexterous robot as a de facto large gripper. During evolution, both the dexterous arm and the elbow free joint gradually shrink to zero so that the dexterous robot eventually evolves to be firmly attached to the target robot end effector.
Our robot evolution solution includes the changing of D independent robot parameters such as body sizes and mass, and joint ranges and damping, where D = 40 for target Sawyer robot and D = 42 for target Jaco robot. The evolution process for Sawyer robot is illustrated in Figure 2.
3.3 Multi-dimension Continuous Robot Evolution for Policy Transfer
As shown in Section 3.2, the evolution from a human-like dexterous robot to a target commercial robot is an extremely high-dimensional problem in robot parameter space (e.g. D 42). This means there exist numerous choices of intermediate robot sequences that connect source to target robots. However, previous work REvolveR [18] assumes the same and uniform evolution progress hard-coded for all robot parameters and could be sub-optimal. We envision that when transferring the policy through intermediate robots, the optimal robot evolution should be flexible and able to smartly bypass difficult robot configurations to reach the target robot efficiently. In this section, we introduce an algorithm for automatically finding such optimized evolution strategy. We formulate the problem as a joint optimization of both robot evolution and the policy.
Problem Statement Suppose the source and target robot morphology is matched using the method in [18] or Section 3.2. Therefore the D kinematies parameters of the source and target robots be mapped to the same space denoted as Os, Or E RR. Continuous function F 10 > .M defines an intermediate robot by interpolation between all pairs of kinematic parameters 0 (1 a) o
Os + a @ OT where each of the D kinematics parameters. Given an expert policy π To(()) on the source robot F(0), the overall goal is to find the optimal policy TF(1) on target robot F(1). Note that the formulation of REvolveR [18] is special case of our problem setting with a- a 1 and [0, 1].
[0, 112 is the evolution parameter that describes the evolution progress of
Differentiable Evolution Path Search (DEPS) We adopt multi-dimensional robot evolution as a solution to the above problem illustrated in Figure 3(a). Suppose there is robot evolution path r == (Foo)) F(an)) F(a2)..... -(x F(ax)) where dg = 0, ar 1. In k-th phase, the policy optimization objective is to maximize the expected reward (Fox.F(ox)] on robot (. For all k; the evolution parameter step size g ||o 0x++1/12 is sufficiently small. In this way, we decompose the difficult robot-to-robot policy transfer problem into a sequence of K easy problems.
The optimal solution to the above problem requires finding the optimal evolution path t together with policy optimization. We propose an algorithm named Differentiable Evolution Path Search (DEPS) to find both the optimized robot evolution path 7 in tandem with policy optimization. Given the current evolution parameter ogg aim to find the next best evolution parameter 03+1 aq +4 as well as its optimal policy π F(a))))) The optimization objective is formulated as
1 2 =al11 (ax + 4
max L. la
L L= = tuu)
(1)
where À + is weighting factor. The first term optimizes the policy reward while the second term encourages the evolved robot to move close to the target robot. Since step size & is small, we assume L is locally differentiable w.r.t. 4. Then the direction of 4 can be estimated by
l VoL= VaBip".F(a))
+ ax) J X(1 ax))
(2)
Q=
where J V.Elp".aaa is the Jacobian of expected reward w.r.t. evolution parameter a. The Jacobian J can be estimated by finite difference with Monte-Carlo sampling of a. Concretely, we sample random vectors d, :2....,6., On with 2-no of &. Suppose do 0. For each 8,,
4

---

Algorithm 1 Multi-dimensione Robot Evolution Policy Transfer
1: Notation Summary:
2: [0, 1J robot evolution parameter: F [0, 1] + M: continuous robot evolution function; *F(o)= expert policy on the source robot; E Rt: evolution step size; E R: reward threshold; R R*: weight factor; À1 (0, 1): shrink ratio;
3: a + 0. T t TF(O), SP) initialization
4: while a * do IEIp
then ō1, .n......., 6n (E) 6u +- // sample 6, vectors execute T and P < p p=. (.)+ Vi E {0..... ... .n]
gP (4)
6:
7:
8:
compute Jacobian J from 6, and P3 according to Eq.(4)
9:
+ J/| 2 A(1 ax 1 al12 I
10:
g
8.
11:
for e O...,.., Nv. do
12:
B Uniform(1 - -1.
sample rollouts and train on robot F(a + (1 end for 15: end if
13:
(b)
14:
Figure 3: (a) Possible robot evolution
paths from source to target robot. (b)
16: a mmiimmaxffaa + +,, 1) make sure stay in [0, 1]
Computation of the next evolution pa- rameter a++1 a2 + 4 in the path.
17: end while
18: return π
execute on robot F(oq + 6:) to get episode reward pi = p(ast()))))) By the definition of Jacobian, if there were no noise, ideally we should have
J J p² F(a -
Pj PO:
Vi
(3)
However, there is usually inevitable noise in the policy rollout and pt Let p [e P2..... ........ Pn. and .2.., 8n). The gradient can be estimated by Least Squares Minimization
(A A (p- po
Since the scale of the episode reward pT. F((a) and therefore J is task- desk-eepnndent and can be arbitrary, stabilize the evolution path search we scale both J and 1 ai in Equation (2) be unit vectors to obtain the direction of evolution progression as follows
4 J|1 + X(1 0x)//111 1
(5)
The next evolution parameter is obtained by 0x-+1 04 + E +/////// where is the evolution step size. The geometric interpretation of Equation (5) illustrated in Figure 3(b).
We point out that the optimization objectives of our DEPS in Equations (2) and (5) can be viewed as a generalization of multiple previous works. When x - +00, the problem reduces to vanilla REvolveR [18] where = a 1; when À - 0, the problem reduces to unrestricted robot and policy joint optimization such as [20] and [21]. Note that when
1, {ax}* guaranteed converge to 1, because it is easy to show for all 1 with -noom of &, || - (ax+1 +1)l2 < ||1 - au - E| 1/.
Policy Optimization along the Evolution Path Given the next evolution parameter ax++1 a₁ + l and the optimized policy TF(a,). the next goal is to find the optimal policy T)(())))) on robot F(a++1). Instead of directly fine-tuning F((ax), on 0l++1 we adopt the following optimization objective with randomized evolution progression similar to [18] to achieve the goal:
= F(ax+
F((()))) arg max
E
3-Unnfomm1 (1111)))
where \ E 1) is the ratio of range shrinkage and N is the iteration counter. The above optimization objective allows the policy to be fine-tuned on a set of randomly sampled robots within range along the Ię to improve the stability and robustness of training while also ensuring the sampled evolution parameter a to converge to ax+1 = ay +4 when e - +oo. In practice, a batch of & in Equation (6) is sampled in each iteration. The overall algorithm is illustrated in Algorithm 1.
5

---

4 Related Work
Learning from Human Demonstration Human demonstrations are often collected in terms of trajectories that could be obtained by teleoperation via virtual reality, motion capture markers, or kinesthetic control of robot. A policy is then learned by cloning the demonstrations [22, 23, 24, 25, 26, 27] or through inverse reinforcement learning [25, 28, 29, 30, 31]. In contrast to these methods, in our approach, the human does not control the robot to which the demonstration is transferred.
Human demonstrations need not always be in the form of state space information and recent line of work rely on visual demonstration to enhance the scalability of data collection as it is much easier to watch humans than ask them to collect demos [32, 33, 34, 35, 36, 37, 38, 39, 40, 41]. Although in our case, the demonstrations are converted to the state space of the dexterous hand robot through inverse kinematics, they are originally collected by watching humans. Different from these prior works, use the demonstration to transfer the policy to very different robots than the demonstrator.
Learning Controllers for New Robot Morphology Evolutionary methods to get new shapes date back to the early days of automata theory from Von Neumman [42]. In early 90s, Karl Sims showed the effectiveness of evolutionary methods in optimizing robot shapes and controllers [43, 44]. These works laid the foundation of the area where not only robots are optimized but also controllers are transferred across evolved robots. In recent years, many works have revived this idea using deep learning based graph evolution [45, 46, 47, 48, 49]. Our work does not treat the morphological evolution in the form of graph networks but rather gradually morphs the source robot to target robot.
Relation to Path Planning Our robot evolution path search problem is similar to the problem of path planning [50, 51, 52, 53, 54, 55]. However, unlike ordinary planning where the obstacles are known, in our problem the difficult robot configurations are unknown. Moreover, in ordinary path planning, evaluating the cost of a path is cheap, while in our case, transferring the policy to new robot requires a large number of policy optimization iterations and is computationally costly.
Experiments
Our experiment is designed to verify the following two hypotheses: (1) our proposed DEPS al- gorithm can find optimized evolution paths in high-dimensional robot parameter space improve policy transfer over vanilla linear evolution path [18]: (2) our HERD framework can learn from human demonstrations recorded in diverse modalities including sensor glove signal and visual data.
Experiment Settings We conduct experiments in MuJoCo physics simulation engine [56]. We adopt the five-finger dexterous hand robot provided in the ADROIT platform [57]. The high-fidelity models of the target robots and grippers are imported from robosuite environment [58] and have been used in previous robotics publications that eventually transferred their experiments to real robots such as [59, 60]. We use NPG [61] as the RL algorithm for policy optimization. Due to the nature of robot-to-robot policy transfer, the total number of RL iterations it takes to reach the goal of a certain success rate or episode reward cannot be set beforehand. So we instead report the number of policy training iterations and simulation epochs needed to reach 80% success rate on the tasks.
5.1 Learning from Sensor Glove Signal Demonstrations
Tasks and Demonstrations We use the three tasks from the Hand Manipulation Suite (HMS) [1]: Hammer, Relocate, and Door In Hammer, the task is to pick up the hammer and smash the nail into the board: In Relocate, the task is to pick up the ball and move it to the target position; In Door, the task is to turn the door handle and fully open the door. We use a sparse reward setting where only task completion is rewarded. For each task, HMS [1] provided 25 human demonstrations recorded as palm and finger locations using a sensor glove named CyberGlove III. Then the expert policy on the dexterous robot is trained using DAPG [1].
Qualitative Results We provide visualization of the policies on the intermediate robots learned during evolution with our HERD in Figure 4. Starting from the dexterous robot, our proposed DEPS algorithm is able to find the next optimized evolved robot. The policy is also able to iteratively adapt to new evolved robots and successfully transfer to the target robots.
Quantitative Comparison with Baselines We compare our HERD against the following baseline methods for learning a policy on the target robot: (1) From Scratch: we train policy on the target robot from scratch with the RI. algorithm under both sparse and dense shaped rewards; (2) DAPG

---

Figure 4: Visualization of policies evolving intermediate robots on Hand Manipulation Suite tasks [1]. We show the results of the learned policies on Door, Hammer and Relocate tasks in three rows respectively. Zoom in for better view.
Reward
Door
Haxmer
Relocate
Agent
#f sim epochs # of train iters
#of sim epochs
# of train iters
#f sim epochs
# of train iters
From Scratch
Shuped
None
608700 1 79410 5U725 6617
248460 9732
20705 I 811 693540 1 118860
57795 1 9905
Sparse
None
DAPG [T]
Sparse
Sawyer
245639 4 24934 20471 2077
161047 + 14027
13420 + 1168 T1433 + 15343
9327 4 1278
REvalveR [18]
Sparce
273663
HERI (Ours)
Sparse
ADROIT
201199 + 23466 TT229 1166
91674 +. 6557
4434 + 608 145138 = 4207
8294 + 1265
Performance with Sawyer robot the target robot.
Reward
Agent
* sim epochs of train iters
of sim epochs of train iters of sim epochs
of train iters
REvalvek [18] Sparse
ADROIT
367612 1 48859 17652 1: 2915
220169 1 20262
11626 = 509 259387
2641
13215
223
HERD (Ours) Sparse
ADROIT
3351% 17938 1338 2616
206214 1 6376 8810 390 214200 5108
10179 704
(b) Performance with Jaco robot as target robot.
Table 1:
Experiments Hand Manipulation Suite tasks [1]. The evaluation metrics is the
number of simulation epochs and policy optimization iterations needed to reach 80% task success rate, shown the "mean + standard deviation" from runs with 5 random seeds.
[1]: a variant of NPG [61] with demonstration -attrrattio--uummeeeeed policy gradient. We use the rollouts of transferred HERD policy the target robots as the demonstrations for DAPG; (3) REvolveR [18]: robot-to-robot policy transfer with continuous and linear robot evolution path which is a special case of our HERD. The results are illustrated in Table 1.
Training RL from scratch never completes the task to receive reward and is not able learn these tasks. Robot-to-robot evolution solutions with sparse rewards are able to successfully transfer the policy and even significantly outperform training from scratch with shaped rewards. Moreover, our HERD outperforms REvolveR [18), though the margin in simulation epochs is smaller than RL iter- ations in that HERD spends additional simulation epochs obtain Jacobian in Equation (4). We notice that our HERD also has advantage over DAPG trained using expert rollouts directly collected the target robot. This shows that training policy while maintaining sufficient reward/success during robot evolution has advantage over exploration from scratch even with demonstration.
5.2 Learning From Visual Demonstrations
Task and Demonstrations In addition to the demonstration captured by sensor glove, interested in showing learning from visual human demonstration with our HERD. We use Dex YCB dataset [2] the source of human demonstration. DexYCB dataset is multi-view RGB-D video dataset capturing human hand manipulating YCB objects [62]. We define the task as grasping and then moving the object of interest to the desired position within distance threshold as demonstrated in the Dex YCB videos. We use 5 YCB objects in Table 2. For each object, we take one video which results in 5 demonstrations in total. The human hand joint poses and object 6D poses estimated by HRNet32 [63] and CozyPose [64] respectively. Then the expert dexterous policy is trained using the method in Section 3. The reward is sparse task completion reward.
also
7

---

Figure 5: Visualization of policies evolving intermediate robots DexYCB dataset manip-
ulations [2]. The target robot is Sawyer. We show the results manipulation policies learned from human demonstration 011.banana m06.ustaard.ottlee and 007 007_tuna fish in three rows respectively.
REvolveK
HERD (Ours)
YCB Objects
of sim opochs
# train ieers
# epochs
#f train iters
006 nustard battle bottle
279363 4: 20115
25643 = 2181
253920 + 22745
19361
1799
007 tuna fish can
320951 + 14466
29812 + 1569
286550 33816
23165
2693
008.pudding.bor
306532 +:
28537
1522
32775
22866
01l.banana
32094 1: 2288
4797 262
31811 1 3490
4022
349
025 bowl
817255 1 46721
73799 = 3892
725919
1 59419
63957
5105
Table 2: Experiments on the DexYCB dataset [2]. The target robot is Sawyer. The evaluation
metrics is the number of simulation epochs and policy optimization iterations needed to reach task success, shown as the "mean standard deviation" from runs with 5 random seeds.
Result Analysis and Comparison We compare against REvolveR [18] and illustrate the experi- ment results Table 2. Our HERD outperforms REvolveR in terms policy iterations as well as most reported simulation epoch metries. One interesting finding that the policy for manipulating 011.banana object is the casiest task among all objects. A possible reason is that 01i_banana object has geometry most suitable for both the dexterous hand and target gripper.
Qualitative Results We visualize the policies on the intermediate robots learned during evolution with our HERD in Figure 5. Again, our proposed DEPS algorithm is able to find the next optimized evolved robot while successfully transferring the policy the target robot. For more details on the visualizations in this section, please refer to the supplementary video.
Conclusion
In this paper. we propose a framework named HERD for learning from human demonstration. HERD framework utilizes the continuous evolution of robots to transfer the expert policy from em- bodied dexterous hands to a target robot. We conduct experiments using the human demonstrations from Human Manipulation Suite [1] and Dex dataset [2]. We show that the proposed approach is able to transfer expert policy trained using human demonstrations in sensor glove signals or visual data target two-finger Sawyer and three-finger Jaco robots, even with challenging sparse rewards.
Limitations We expect that our HERD would fail in learning from human demonstrations for robots with extremely different functions than humans, e.g. learning object manipulation policy for robot arm mounted with a suction cup from human hand manipulation demonstrations. This is because proposed human-to-robo continuous evolution solution only handle different morphology but cannot handle different functions of the robot end-effector. We leave the exploration more continuous evolution solutions future work.
general

---

Acknowledgement This work is in part funded by JST AIP Acceleration, Grant Number JP- MJCR20U1, Japan. Deepak Pathak is supported by NSF grant IIS-2024594.
References
[1] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstra- tions. In Proceedings of Robotics: Science and Systems (RSS), Pittsburgh, Pennsylvania, June 2018. doi:10. 55077/SSS 201 XIVV.09.
[2] Y.-W. Chao, W. Yang. Y. Xiang, P. Molchanov, A. Handa, J. Tremblay, Y. S. Narang, K. Van Wyk, U. lqbal, S. Birchfield. J. Kautz, and D. Fox. DexYCB: A benchmark for capturing hand grasping of objects. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[3] B. D. Argall, S. Chernova, M. Veloso, and B. Browning. survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469-45) -866, 2009.
[4] D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti,
J. Munro, T. Perrett, W. Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the European Conference on Computer Vision (ECCV), pages 720-736, 2018.
[5] K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, M. Martin, T. Nagarajan, I. Radosavovic, S. K. Ramakrishnan, F. Ryan, J. Sharma, M. Wray, M. Xu, E. Z. Xu, C. Zhao, S. Bansal, D. Batra, V. Cartillier, Crane, T. Do. M. Doulaty, A. Erapalli, C. Feichtenhofer, A. Fragomeni, Q. Fu, C. Fuegen, A. Ge- breselasie, C. Gonzalez, J. Hillis, X. Huang, Y. Huang, W. Jia, W. Khoo, J. Kolar, S. Kottur, A. Kumar, F. Landini, C. Li, Y. Li, Z. Li, K. Mangalam, R. Modhugu, J. Munro, T. Murrell, T. Nishiyasu, W. Price, R. Puentes, M. Ramazanova, L. Sari, K. Somasundaram, A. Souther- land, Y. Sugano, R. Tao, M. Vo. Y. Wang, X. Wu, T. Yagi. Y. Zhu, P. Arbelaez, D. Crandall, D. Damen, G. M. Farinella, B. Ghanem,
Joo, K. Kitani, H. Li, R. Newcombe, A. Oliva, H. S. Park, J. M. Rehg. Y. Sato, J. Shi, M. 7. Shou, A. Torralba, I.. Torresani, M. Yan, and Malik. Ego4d: Around the world 3,000 hours of egocentric video. arXiv preprint arXiv:21 0.07058, 2021.
K. Ithapu, C. V. Jawahar,
[6] S. Hampali, M. Rad, M. Oberweger, and V. Lepetit. Honnotate: A method for 3d annotation of hand and object poses. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11996-2266 2020.
[7] T. Kwon, B. Tekin, J. Stühmer, F. Bogo, and M. Pollefeys. H2o: objects for first person interaction recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10138-10148, 2021.
Two hands manipulating
[8] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured predic- tion to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Pro- ceedings, 2011.
[9] A. Y. Ng. S. Russell, et al. Algorithms for inverse reinforcement learning. In Ieml, volume 1, page 2, 2000.
[10] I. Radosavovic, X. Wang, L. Pinto, and J. Malik. State-only imitation learning for dexterous manipulation. arXiv preprint 20X4::2004.04650, 2020.
[11] H. Xiong. Q. Li, Y.C. Chen, H. Bharadhwaj. S. Sinha, and A. Garg. Learning by watching: Physical imitation of manipulation skills from human videos. arXiv preprint arXiv:2101007244 2021.
[12] A. Bajesy, D. P. Losey, M. K. O'Malley, and A. D. Dragan. Learning robot objectives from physical human interaction. Conference on Robot Learning, pages 217-226. PMLR, 2017.
[13] M. Li, A. Canberk, D. P. Losey, and D. Sadigh. Learning human objectives from sequences of physical corrections. 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 2877-2883. IEEF, 2021.

---

[14] A. Mandlekar, Y. Zhu, A. Garg, J. Booher, M. Spero, A. Tung, J. Gao, J. Emmons, A. Gupta, E. Orbay, al. Roboturk: crowdsourcing platform for robotic skill learning through imita- tion. In Conference on Robot Learning, pages 179-893, PMLR, 2018.
[15] Y. Qin, H. Su, and X. Wang. From one hand to multiple hands: Imitation learning for dexterous manipulation from single-camera teleoperation. arXiv preprint arXiv:2204. 12490, 2022.
[16] Y. Qin, Y.-H. Wu, S. Liu, H. Jiang, R. Yang. Y. Fu, and X. Wang. Dexmv: Imitation learning for dexterous manipulation from human videos. arXiv preprint arXiv: 058777 2021.
[17] F. Liu, Z. Ling, T. Mu, and H. Su. State alignment-based imitation learning. arXiv preprint OrXiv:/9/1 1. 10947, 2019.
[18] X. Liu, D. Pathak, and K. M. Kitani. Revolver: Continuous evolutionary models for robot-to- robot policy transfer. arXiv preprint arXiv:2202.052444 052444, 2022.
[19] V. Kumar and Todorov. Mujoco haptix: A virtual reality system for hand manipulation. In 2015 JEEE-RAS 15th International Conference on Humanoid Robots (Humanoids), pages 657-663 2015. doi:10. Noi:10.1199/HUI 10//H//UAAANODID
[20] X. Pan, A. Garg, A. Anandkumar, and Y. Zhu. Emergent hand morphology and control from optimizing robust grasps of diverse objects. In 2021 IEEE International Conference on Rebotics and Automation (ICRA), pages 7540-7547. IEEE, 2021.
[21] Y. Yuan, Y. Song, Z. Luo, W. Sun, and K. Kitani. Transform2act: Learning a transform-and- control policy for efficient agent design. arXiv preprint arXiv:2110.03653 2021.
[22] D. A. Pomerleau. ALVINN: An autonomous land vehicle in neural network. In NIPS, 1989.
[23] B. D. Argall, S. Chermova, M. Veloso, and B. Browning. survey of robot learning from demonstration. Robotics and autonomous systems, 2009.
[24] S. Schaal. Is imitation learning the route to humanoid robots? Trends in cognitive sciences, 1999.
[25] A. Y. Ng and S. Russell. Algorithms for inverse reinforcement learning. In ICML, pages 663-670, 2000.
[26] B. Akgun, M. Cakmak, J. W. Yoo, and A. L. Thomaz. Trajectories and keyframes for kines- thetic teaching: A human-robot interaction perspective. In HRI, March 2012.
[27] T. Zhang, Z. McCarthy, Jow, D. Lee, K. Goldberg, and P. Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. CoRR, abs/1710.04615. 2017.
[28] P. Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In JCML, 2004.
[29] B. D. Ziebart, A. Maas, J. A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforcement learning. In AAAI, 2008.
[30] S. Levine, P. Pastor, A. Krizhevsky, and D. Quillen. Learning hand-eye coordination for robotic grasping with large-scale data collection. In ISER, 2016.
[31] J. Ho and S. Ermon. Generative adversarial imitation learning. In NIPS, 2016.
[32] H. Koichi and H. Tom. Visual servoing: real-time control of robot manipulators based on visual sensory feedback, volume 7. World scientific, 1993.
[33] B. H. Yoshimi and Allen. Active uncalibrated visual servoing- In Robotics and Au- tomation, 1994. Proceedings,, 1994 IEEE International Conference on, pages 156-161. IEEE, 1994.
[34] W.J. Wilson, C. W. Hulls, and G. S. Bell. Relative end-effector control using cartesian position based visual servoing. IEEE Transactions on Robotics and Automation, (655)66846966 1996.
10

---

[35] G. Caron, E. Marchand, and E. M. Mouaddib. Photometric visual servoing for omnidirectional cameras. Autonomous Robots, 35(2-3):177-193, 2013.
[36] P. Sermanet, C. Lynch. Y. Chebotar, J. Hsu, E. Jang, S. Schaal, and S. Levine. Time-contrastive networks: Self-supervised learning from video. In ICRA, 2018.
[37] P. Sharma, D. Pathak, and A. Gupta. Third-person visual imitation learning via decoupled hierarchical controller. arXiv preprint
2019.
[38] Y. Liu, A. Gupta, P. Abbeel, and S. Levine. Imitation from observation: Learning to imitate behaviors from raw video via context translation. ICRA, 2018.
[39] L. Smith, N. Dhawan, M. Zhang, P. Abbeel, and S. Levine. Avid: Learning multi-stage tasks via pixel-level translation of human videos. In RSS, 2020.
[40] D. Pathak, P. Mahmoudieh, G. Luo, P. Agrawal, D. Chen, Y. Shentu, E. Shelhamer, J. Malik, A. A. Efros, and T. Darrell. Zero-shot visual imitation. In ICLR, 2018.
[41] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-imane translation using cycle- consistent adversarial networks. ICCV, 2017.
[42] Von Neumann, A. W. Burks, et al. Theory of self-reproducing automata. IEEE Transactions on Neural Networks, 1966.
[43] K. Sims. Evolving virtual creatures. In Computer graphics and interactive techniques, 1994.
[44] K. Sims. Evolving 3d morphology and behavior by competition. Artificial life, 1994.
[45] D. Pathak, C. Lu, T. Darrell, Isola, and A. Efros. Learning to control self self-assembling morphologies: study of generalization via modularity. NeurlPS, 2019.
[46] Wang, R. Liao, J. Ba, and S. Fidler. Nervenet: Learning structured policy with graph neural networks. ICLR, 2018.
[47] T. Wang, Zhou, S. Fidler, and J. Ba. Neural graph evolution: Towards efficient automatic robot design. arXiv preprint 1906.053000 2019.
[48] D. Hejna III, P. Abbeel, and L. Pinto. aask-agnostic morphology evolution. arXiv preprint arXiv:2102.13100 2021.
[49] A. Gupta, S. Savarese, S. Ganguli, and L. Fei-Fei. Embodied intelligence via learning and evolution. arXiv preprint Siii:2200.0220022 2021.
[50] S. LAVALLE. Rapidly-exploring random trees: A new tool for path planning. Computer Science Dept. Oct., 98(11) 1), 1998.
[51] S. Karaman and F. Frazzoli. Sampling-based algorithms for optimal motion planning. The international journal of robotics research, 30(7):846-894, 2011.
[52] F. Islam, J. Nasir, U. Malik, Y. Ayaz, and O. Hasan. Rrta-smart: Rapid convergence implemen- tation of rrt towards optimal solution. In 2012 IEEE international conference on mechatronics and automation, pages 1651-1656. IEEE, 2012.
[53] A. Perez, R. Platt, G. Konidaris, L. Kaelbling, and T. Lozano-Perez. Lqr-m:* Optimal sampling-base based motion planning with automatically derived extension heuristics. In 2012 IEEE International Conference on Robotics and Automation, pages 2537-2542. IEEE, 2012.
[54] P. E. Hart, N. J. Nilsson, and B. Raphael. A formal basis for the heuristic determination of minimum cost paths. IEEE transactions on Systems Science and Cybernetics, 4(2):100-107, 1968.
[55] S. Koenig, M. Likhachev, and D. Furcy. Lifelong planning a², Artificial Intelligence, 155(1-2): 93-146, 2004.

---

[56] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026- 5033. IEEE, 2012.
[57] V. Kumar, Z. Xu, and E. Todorov. Fast, strong and compliant pneumatic actuation for dexterous tendon-driven hands. In 2013 IEEE international conference on robotics and automation, pages 1519 IEEE, 2013.
[58] Y. Zhu, J. Wong, A. Mandlekar, and R. Martin-Martín. robosuite: A modular simulation framework and benchmark for robot learning. In arXiv preprint arXiv:2009. 12293, 2020.
[59] W. Zhou and D. Held. Learning to grasp the ungraspable with emergent extrinsic dexterity. In ICRA 2022 Workshop: Reinforcement Learning for Contact-Rice Manipulation, 2022.
[60] Z. Jiang, C.-C. Hsu, and Y. Zhu. Ditto: Building digital twins of articulated objects from interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5616-5626, 2022.
[61] A. Rajeswaran, K. Lowrey, E. Todorov, and S. Kakade. Towards generalization and simplicity in continuous control. arXiv preprint
2017.
[62] Y. Xiang, Schmidt, V. Narayanan, and D. Fox. Posecnn: convolutional neural network for 6d object pose estimation in cluttered scenes. arXiv preprint arXiv:1711.00110 2017.
[63] A. Spurr, U. lqbal, P. Molchanov, O. Hilliges, and J. Kautz. Weakly supervised 3d hand pose estimation via biomechanical constraints. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVII 16, pages 211-228. Springer, 2020.
[64] Y. Labbé, J. Carpentier, M. Aubry, and J. Sivie. Cosypose: Consistent multi-view multi-object 6d pose estimation. In European Conference on Computer Vision, pages 574-591. Springer, 2020.
12


Reviews:
[{"review_id": "mPRM3sjYGPW", "reviewer": "Reviewer_bvcA", "summary": "This paper proposes a method for transferring human skills in hand tool manipulation to a simple two-fingered robot. Specifically, a robot model with the same degree of freedom as a human first learns tool manipulation from a human demonstration, and then the robot model is gradually transformed into a two-fingered robot through an iterative process of reducing degrees of freedom and relearning policies. The main contributions of this study are the construction of a framework that integrates the process from human demonstration to tool manipulation by a two-fingered robot, and the reduction in the number of simulations required for training compared to existing methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The most significant contribution of this paper is that it provides a architecture from a human demonstration to task execution by a simple two-fingered robot. On the other hand, the weakest point of the paper is that the experiments were not conducted on a real robot.\n- The tool manipulation task tested in the experiment has complex dynamics, especially contact and friction, which are difficult to simulate accurately and deviate significantly from the real world. The proposed method uses a simulator to learn policies, and it is unclear how well the learning results will perform in the real world.\n- The evolutionary learning of the robot model is based on the existing method REvolveR [18]. The authors state that larger variation in evolutionary paths results in higher efficiency of learning, but they do not provide specific experimental data to support this.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The most significant contribution of this paper is that it shows a architecture from a human demonstration to task execution by a simple two-fingered robot, and it deserves to be accepted. However, due to the lack of mathematical novelty and the lack of real robot experiments, the evaluation was lowered one rank.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "- Compared to the existing method REvolveR [18], it is not intuitive that the total number of simulations is reduced despite the increase in the dimensionality of the parameters involved in the evolution of the robot. In Table 1 and 2, the number of simulations used for policy learning and Jacobian computation should be broken down clearly and compared with REvolveR. The definition of simulation epoch should also be clarified.\n- The task performance should be higher for hands with higher degrees of freedom. Therefore, in Eq.2, the Jacobian term (first term) is oriented toward the original dexterous robot hand. On the other hand, as the authors said, the lambda term (second term) is oriented toward the two-fingered hand. It is thought that there is a risk of them cancelling each other and stagnating in the middle of the learning process. Please add an explanation for this question.\n"}}, {"review_id": "l1MF7VEYqhW", "reviewer": "Reviewer_WPK2", "summary": "This work introduces HERD, a framework for learning robotic policies from human demonstrations where the target robot has a different morphology (e.g. 2 fingered gripper) to the human (e.g. a hand). The framework has two main components. (1) Learning a robotic policy from human demonstrations where the robot policy has a similar morphology to the human demonstrations and (2) transfer the learned robotic policy to a target robot with a different morphology through continuous evolution.\n\nThe main contributions of this paper are as follows\n* HERD, a framework for learning from human demonstrations by continuously evolving from source to target robots\n* DEPS (Differentiable Evolution Path Search) which is a method for searching in the evolution parameter space to evolve from source to target robot\n* Experiments in simulation which demonstrate that HERD can be used to transfer human demonstrations with sensor or visual observations to a target robot with a two-finger gripper.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* This work address a relevant and important problem setting — how to learn from human demonstrations whilst also able to deploy the learned policies on robots with different morphologies.\n* Simulation experiments are thorough, with relevant baselines\n   * The authors test DEPS on 8 different tasks spanning two datasets and two state spaces (glove signals and visual demonstrations).\n* The authors demonstrate that HERD can successfully be used to evolve policies from visual human demonstrations (the experiments on the DexYCB dataset). To my knowledge this is novel.\n* The idea to use the Jacobian of the expected reward with respect to the evolution parameter to determine the evolutionary path is interesting.\n* The paper is clearly written, and DEPS is well explained\n* Related work appears good with one exception noted below\n* The supplementary video is nice. I appreciate the time and effort the authors made to prepare a visual summary of the work. I especially appreciate the numerous examples of policies at different stages of the evolutionary process solving the different tasks.\n\nWeaknesses\n* I am not convinced that DEPS improves sample efficiency over REvolveR\n   * In all cases the number of sim epochs to reach the same performance appears similar to REvolveR when compared with the reported standard deviation for DEPS.\n      * I note that it is possible that the means have non overlapping confidence intervals and are significantly different.\n   * I don’t think number of training iterations is a relevant metric for comparing policies since algorithms may differ substantilly in what occurs during a training iteration. If the authors intend to give a measure of training time I suggest using FLOPS or wall clock training time (although this is more relevant for a physical robot).\n* The authors present a framework (HERD) but only demonstrate it on one source and target robot. I would be more convinced of the generality of their framework if they had demonstrated HERD on multiple target robots, especially given that evolution from the source to target robot in this paper appears to have been previously demonstrated in REvolveR.\n* The learned policies are only deployed in simulation and there is no discussion of a path to deployment on a physical robot\n* I appreciate the authors mentioning in the limitations section that HERD might fail if the target morphology was very different from the demonstrations. However, the limitations section is very brief, only 3 lines long. I would have liked to see more of a discussion as to why it might fail (e.g. is it impossible to define a continuous evolutionary path? Is it impossible to define MDPs with the same action space for source and target robots?). It would also have been really nice if this failure case could have been demonstrated experimentally.\n* Limited algorithmic novelty and only moderate empirical novelty\n   * HERD/DEPS is quite similar to REvolveR.\n   * I note that the “core idea is to define a continuous interpolation between the dexterous hand robot and a target robot that have a different morphology” (line 41-42) was the main idea of prior work REvolveR.\n   * Furthermore, successful evolution between the source and target robot has been demonstrated in prior work (REvolveR).\n   * One way to improve the paper could be to provide a more in depth analysis of how DEPS differs from REvolveR. For example, does DEPS ever result in a noticeably different evolutionary path to REvolveR? Are there certain joints it favors vs REvolveR? Are there any circumstances under which REvolveR fails but DEPS succeeds? As of now it is not clear why I would choose to implement the more complex DEPS over REvolveR. \n* Given that DEPS is similar to REvolveR and appears to build off it, I would have liked to see an explicit comparison with revolver in the related work.\n   * The authors do cite REvolveR in the preliminaries, however REvolveR is absent from the related work\n   * Additionally, in the last example in the supplementary video (tuna_fish_can object) where alpha is shared across all evolutionary parameters the authors refer to this as a special case of their framework. As I understand it, this is REvolveR and I think this should be acknowledged in the video.\n   * Similarly HERD appears to be mostly a combination of existing techniques (DAPG / CozyPose + HRNet + Curriculum learning plus REvolveR) with a small modification to the Revolver algorithm. Combining pre-existing techniques can be a valuable contribution in itself, however I would have liked to see this more explicitly stated.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper addresses a relevant and interesting problem in robot learning, and the authors demonstrate the effectiveness of HERD on a number of different tasks in simulation. However at the moment the weaknesses outweigh the strengths of the paper.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "* Please add a discussion about how this work could be used to deploy policies on a physical robot.\n* Please elaborate on the limitation raised\n* Please address the points raised regarding referencing and comparing with REvolveR\n* There are a number of grammatical errors. I give some examples below. Please review the paper for grammar.\n   * Line 21: paradigm → paradigms\n   * Line 83: demonstration → demonstrations\n   * Line 84: these → this\n   * Line 85: on → on a\n   * Line 87: in → in a\n   * Line 88: as → as in\n   * Line 89: policy → policies\n   * Line 90: demonstration → demonstrations\n   * Line 240: “never completes the task to receive reward” this doesn’t make sense.\n   * Line 246: training policy → training a policy\n"}}, {"review_id": "O3Ow36Sw7dW", "reviewer": "Reviewer_Pofh", "summary": "The authors present an approach for learning robot manipulation tasks from human demonstrations (HERD). Since the embodiments of the human demonstrator and that of the robot differ, the demonstration data cannot be utilized directly for robot learning. For overcoming this issue, a reinforcement learning agent learns to control a virtual robot which is step by step morphed from a human representation into the final robot representation. The agent's learning process is bootstrapped by using the demonstration data with the human embodiment. Subsequently, the embodiment is morphed gradually towards that of the robot. Every micro-update of the embodiment is followed by fine-tuning the agent's policy on the new embodiment. The main contribution of the paper is an algorithm for deciding how to shape the evolution path towards the final robot embodiment based on a Jacobian approximation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The authors show that their approach outperforms existing approaches due to the added freedom in adapting the embodiment parameters (even though the improved performance compared to REvolveR is rather small)\n- The paper is written well and it is straightforward to read and understand\n\nWeaknesses\n- The approach is only applied to a single robot (Baxter). I think the paper would have been a lot stronger if the authors showed that multiple robots with different embodiments can successfully learn a task from the same human demonstration data.\n- The tasks used for the experimental evaluation are very \"kinematic\". The authors do not discuss in the paper if the approach could also be applied to more \"dynamic\" tasks.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, I think it is a solid paper without bigger weaknesses. However, I am not sure if the contribution of the paper justifies a publication at the top-tier CoRL conference. The improvements compared to previous state-of-the-art approaches (mainly REvolveR) seems to be rather small. Compared to REvolveR, the RL training seems to converge quicker to an acceptable task success rate (80% in the experimental evaluation). There is no empirical evidence that it also outperforms this algorithm. As mentioned in the other sections, the paper could be improved by applying the method to multiple different robots. Also, an ablation study would be helpful to justify design choices (for instance to evaluate the usefulness gradient computation, see my last point in the Issues-Section)", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "- Preliminaries: Here, you explain how REvolveR works, but at this point it is unclear how the method is connected to your approach. Therefore, the reader is wondering what the purpose of the explanation is. The connection to the method becomes clear in Section 3.3, but is confusing in Section 2. I recommend either adding a sentence (e.g., like \"we extend this method\" / \"our approach is based on REvolveR\") or moving the paragraph to another Section.\n\n- Line 84: \"These research progress\" --> This research progress?\n\n- Line 85: Figure 1(1)(2) looks strange. Just use Figure 1. The reader should be smart enough to read the caption and figure out on his own that only parts of the figure are relevant here.\n\n- Regarding the approach, I was wondering if computing/approximating the gradient is actually necessary. Since you sample new embodiment parameters and do rollouts with each new parameter, why don't you simply pick the parameter set that yields the hightest reward instead of merging all parameter sets into a gradient based on the rewards? Referring to Figure 3, why not picking the best red arrow instead of computing a direction from all red arrows? If computing the Jacobian has advantages here it would be good if the authors could comment on it. It would also be possible to add this strategy as a baseline to the experimental validation."}}, {"review_id": "AttBnMyvfeA", "reviewer": "Reviewer_mLRs", "summary": "The paper addresses the issue of robot learning from human demonstration with different structures. The proposed approach (named HERD) evolves a human hand into a robot hand, by simulating the evolutions in a dynamics simulator such as Gazebo. A policy is first learned from human demonstrations, and is transferred to the target robot through training on a sequence of intermediate simulated robots that evolve from the human hand to the target robot. At each evolution stage, the policy is updated and adapted to the new evolved hand by interacting in the simulator. To achieve this, the paper proposes to jointly optimise searching for optimal robot evolution paths and policy learned at each evolution.\n\nThe paper builds on top of a recently published work (REvolveR) using the same idea but with fixed hyper-parameters for guiding the search for evolution paths. Their extension to this approach performs significantly better on two benchmarks.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nHERD (and REvolveR) avoid the need statistical matching approaches that require the source and target to have similar kinematics and dynamics structures. \n\nWeakness\n\nWhile it doesn't require a human to match the state and action spaces of the source and target, it still requires human effort to \"match the morphologies\" of the two and \"design a shared high-dimensional robot parameter space\". The design of the robot parameter space still requires as much robotics expertise as the 'demonstration conversion' techniques the paper criticises. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Experimental results are not showing significant gains compared to REvolveR. In some cases REvolveR even performs better. This is in addition to the criticism I gave earlier, about the approach requiring expertise to match the morphologies and designing the shared robot parameter space. And this approach seems to be human-robot pair specific.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "1. How general is the matching process? For example, how easily can it be adapted to a 3-finger robot or even another 2-finger robot other than the Sawyer robot? In which case it should be easy to demonstrate on such robots since experiments are performed in simulations.\n2. The approach would be stronger if instead of keeping the thumb and first finger and evolving the rest, we could randomly choose any 2 fingers to keep fixed and evolve the rest. At the very least an ablation study on this would shed light on the generalization of HERD."}}]

Author rebuttal:

Decision: Accept (poster)