Meta-Evolve: Continuous Robot Evolution for One-to-many Policy Transfer
Authors: Xingyu Liu
Deepak Pathak
Ding Zhao
Venue: ICLR 2024 poster (2024)
Keywords: policy transfer
transfer learning
imitation learning
reinforcement learning
Primary area: reinforcement learning
TL;DR: A method for efficiently transferring an expert policy from one robot to multiple different robots

Abstract: We investigate the problem of transferring an expert policy from a source robot to multiple different robots. To solve this problem, we propose a method named *Meta-Evolve* that uses continuous robot evolution to efficiently transfer the policy to each target robot through a set of tree-structured evolutionary robot sequences. The robot evolution tree allows the robot evolution paths to be shared, so our approach can significantly outperform naive one-to-one policy transfer. We present a heuristic approach to determine an optimized robot evolution tree. Experiments have shown that our method is able to improve the efficiency of one-to-three transfer of manipulation policy by up to 3.2$\times$ and one-to-six transfer of agile locomotion policy by 2.4$\times$ in terms of simulation cost over the baseline of launching multiple independent one-to-one policy transfers. Supplementary videos available at the project website: https://sites.google.com/view/meta-evolve.

Published as a conference paper at ICLR 2024
EETA-EVOLVE: CONTINUOUS ROBOT EVOLUTION FOR ONE-TO-MAN TO-MANY POLICY TRANSFER
Xingyu Liu, Deepak Pathak, Ding Zhao Camegiie Mellon University (xing Exingyu13, dpathak, dingzhao @andrew. cmu edu
Source Robot
Target Robot I
Target Robot
Source Robot
Meta Robot
Target Robot
Target Robet 2
Target Robot
Target
(b)
(a) REvolveR and HERD (Liu al., 2022a;b) are methods transferring policy between pair of robots using continuous robot evolution. Therefore, to transfer policy the source robot to multiple target robots, they must launch multiple independent runs for cach target robot. (b) Our Meta-Evolve uses continuous robot evolution to transfer expert policy from the source robot each target robot through evolution tree defined by the connections multiple "meta robots", i.c. tree-structured evolutionary robot sequences.
Figure 1:
ABSTRACT
We investigate the problem of transferring an expert policy from source robot multiple different robots. To solve this problem, we propose a method named Meta-Evole that uses continuous robot evolution to efficiently transfer the policy to each target robot through set of tree-structured evolutionary robot sequences. The robot evolution tree allows the robot evolution paths to be shared, so approach can sigmificantly outperform naive one-to-one policy transfer. We present heuristic approach to determine optimized robot evolution tree. Experiments have shown that our method is able to improve the efficiency of one-to-three transfer of manipulation policy by up 3.2x and one-to-six transfer of agile locomotion policy by 2.4x in terms of simulation cost over the baseline of launching multiple independent one-to-one policy transfers. Supplementary videos available the project website: https: /ttp//sttssss/eee google com/ vwt///ieii//// www/e///teeeeeeeeeeeeeeeeeeeeeee mwt.mmmtaaaeeeeeeeeeeeee
INTRODUCTION
The robotics industry has designed and developed a large number of commercial robots deployed in various applications. How efficiently learn robotic skills on diverse robots in scalable fashion? A popular solution is to train policy for every new robot every new task from scratch. This is not only inefficient terms of sample efficiency but also impractical for complex robots due large exploration space. Inter-robot imitation by statistic matching methods that optimize to match the distribution of actions (Ross et al., 2011), transitioned states (Liu et al., 2019; Radosavovic et al., 2020), or reward (Ng et al., 2000; Ho & Ermon, 2016) could be possible solutions. However, they can only be applied to robots with similar dynamics to yield optimal performance.
Recent advances in iooluutionbbased imitation learning (Liu et al., 2022a;b) inspire us to view this problem from the perspective of policy transferring from one robot to another. The core idea is to interpolate two different robots by producing large number of intermediate robots between them which gradually evolve from the source robot toward the target robot. These continuously and gradually evolving robots act the bridge for transferring the policy from the source to the

---

Published as a conference paper at ICLR 2024
target robot. The source robot is usually selected robot such that it is easy to collect sufficient demonstrations to train a high-performance expert policy, e.g. a Shadow Hand robot that can be trained from large-scale human hand demonstration data (Grauman et al., 2022; Damen et al., 2018). While continuous robot evolution has shown success in learning challenging robot manipulation tasks (Liu et al., 2022a), the policy transfer is limited to being between pair of robots. As illustrated in Figure (a), for N different target robots where N > 1, it requires launching N independent runs of robot-to-robot policy transfer and is not scalable. How can one efficiently transfer well-trained policy from one source robot to multiple different target robots?
The history of biodiversity provides inspirations for this problem. In the biological world, similar creatures usually share the same ancestors in their evolution history before splitting their ways to form diverse species (Darwin, 1859). The same holds true for the robotic world. When robots are designed to complete certain tasks, they often share similar forms of morphology and dynamics to interact with other objects in similar ways. Examples include robot grippers that are all designed to close their fingers to grasp objects and multi-legged robots that are all designed to stretch their legs for agile locomotion. Therefore, to transfer the policy to N different target robots, it may be possible to find common robot "ancestors" and share some parts of the robot evolution paths among the target robots before splitting their ways to each target robot. In this way, the cost of exploration and training during policy transfer can be significantly reduced. The idea is illustrated in Figure 1(b).
We propose a method named Meta-Evolve to instantiate the above idea. Given the source and multiple target robots, our method first matches their kinematic tree topology. This allows the source robot and all target robots to be represented in the same high-dimennional continuous space of physical parameters and also allows generating new intermediate robots. To share the evolution paths among multiple target robots, it requires defining a set of robot evolution sequences that are organized in a tree structure with the source robot being the root node. We propose a heuristic approach to determine the robot evolution tree within the parameter space by minimizing the total cost of training and exploration during policy transfer. We formulate the problem as finding a Steiner tree (Steiner, 1881; Gilbert & Pollak, 1968) in the robot parameter space that interconnects the source and all target robots. Our algorithm then decides evolution path splitting based on the evolution Steiner tree.
We showcase our Meta-Evolve on three Hand Manipulation Suite manipulation tasks (Rajeswaran et al., 2018) where the source robot a five-finger dexterous hand and the target robots are three robot grippers with two, three, and four fingers respectively. Our Seta-Evolve reduces the total number of simulation epochs by up to 3.2x compared to pairwise robot policy transfer baselines (Liu et al., 2022a;b) to reach the same performance the three target robots. When applied to one-to-six policy transfer on four-legged agile locomotion robots, our Meta-Evolve can improve the total simulation cost by 2.4x. This shows that our Meta-Evolve allows more scalable inter-robot imitation learning.
PRELIMINARY
Notation We use bold letters denote vectors. Specially, and are the all-zero and all-one vectors with proper dimensions respectively. We use @ to denote the element-wise product between vectors. We use f with subscripts to denote robot physical parameters and a and /3 with subscripts to denote evolution parameters. We use MAX(-) and MIN(-) to denote wlement-wise maximum and minimum of set of vectors respectively. || llp denotes the LP vector norm. denotes the set cardinality.
MDP Preliminary We consider continuous control problem formulated as Markov Decision Process (MDP). It is defnnd by a tuple (S.A, R, 7). where S C R is the state space, A C R is the action space, T S x A + S is the transition function, R S x A + R is the reward function, and 7 E [0, 1] is the discount factor. A policy π: S maps a state to an action where optimal policy TM on MDP M is the one that maximizes the expected value of pT..
T(ls)) is the probability of choosing action at state Suppose M is the set of all MDPs and T.. Ei RR(ss, ag) is the episode discounted reward with policy π on MDP M e M. The
REvolveR and HERD Preliminary Liu et al. (2022b) proposed technique named REvolveR for transferring policies from one robot to a different robot. Given a well-trained expert policy π on a source robot Ms E M, its goal is to find the optimal policy TMM on another target robot M, M. The core idea is to define a sequence of intermediate robots through linear interpolation of physical parameters and sequentially fine-tune the policy on each intermediate robot in the sequence. Liu et al.
2

---

Published as a conference paper at ICLR 2024
(2022a) proposed HERD, which extends the idea to robots represented in high- -igh-dimensional parameter space, and proposes to optimize the robot evolution path together with the policy. Concretely, given source and target robots Ms. Mr E MM that are parameterized in D D-dimensional space. HERD defines a continuous function F : [0, 1j2 + M where F(O) = Ms. F(1) = Mr. Given the physical parameters of the source and target robots Os. Or TR. respectively, for any evolution parameter [0, _] F(a) defines an intermediate robot whose physical parameters are 0 = (L-x)00s)))) ar & Then an expert policy To()) on the source robot F() is optimized by sequentially interacting with each intermediate robot in the sequence F(az)) F(ax)..... F((ok) where æ = 1, until the policy is able act (near) optimally on each intermediate robot. At robot F(oxx), the optimization objective for finding the next best intermediate robot F(04+1) = F(ax; + [x) is
Elp F... 4x) 14] 2/2 ||1 (ac + 44)1
max ||6|||25
max
which optimizes both the expected reward Esp F(au and the L distance to the target robot [1 (a& 4x 122. For all k, the evolution step size ||x112 is small enough so that each policy fine-tuning step is a much easier task. The idea is illustrated in Figure (a))
3 ONE-TO-MANY ROBOT-TO-ROBOT POLICY TRANSFER
3.1 PROBLEM STATEMENT
We investigate a new problem of transferring an expert policy from one source robot to multiple target robots. Formally, we consider source robot Ms E .M and N target robots 1T,1, Mr......., MT.N E M respectively. We assume the state space S, and action space A, reward function R. and discount factor 7 of Ms and all Mr.; are shared and the difference is their transition dynamics T_ Given a well-trained expert policy Ti, on a source robot Ms, the goal is to find the optimal policies Mr on each of the target robot Mr.-- We would like to investigate using the information in iMM to improve the learning of Mr, and study how the learning of each individual Myr can help each other.
We approach this problem by defining multiple meta robots MMaa, E . that shares the same state and action space as Ms and all My,,, The meta robots MMeew are designed such that MMem, interconnects Ms and all Mr.; in an efficient way. Therefore, instead of repeating the process of one-to-one policy transferring for N times, we can transfer source robot policy яş by going through the interconnection formed by meta robots MMen. to reach each individual MT.j.
3.2 MULTI-ROBOT MORPHOLOGY MATCHING AND INTERMEDIATE ROBOT GENERATION
Our problem setting and proposed solution based robot Ms and target robots Mf,; are different in action and state spaces, they can still be mapped the same state and action space based on which the intermediate robots can be defined. The assumption is true for pair of robots as shown in Liu et al. (2022a) and Liu et al. (2022b) where the intermediate robots are produced by robot morphology matching and kinematic interpolation. In this subsection, we show that this assumption can be extended to more than two robots.
assumption that even if the source
Kinematic Tree Matching The topology of the kinematic tree of robot describes the connection of the bodies and joints and reflects the kinematic behavior of the robot. Given two different robots with different kinematic tree, it has been shown in Liu et al. (2022b) that their kinematic trees can be matched by adding extra nodes and edges. This step can be extended to N robots when N > 2. As illustrated in Figure 2(a), by matching proper root and leaf nodes, the matched kinematic tree is essentially a graph union of the kinematic trees of all N robots. This means each robot needs to create additional bodies, joints and motors, though they may be all zero in numbers at the beginning.
The above kinematic tree matching process can be automated by algorithm that achieves the best matching of nodes and edges across N robots. However, in practice, we would like the matching process include reasonable human intervention with enough robotics knowledge, e.g. matching human hand fingers to robot gripper fingers such that the knuckle joints are matched correctly.
Physical Parameter Interpolation After kinematic tree matching, the state and action space of the robots are matched. The difference in the robot transition dynamics is now only due to the differences in physical parameters, such as shapes and mass of robot bodies, gain and armature of joint motors,
3

---

Published as a conference paper at ICLR 2024
Robot
Robot 3
Morphology Matched Robot
Rohot
(a)
(b)
Figure 2: (a) Morphology matching multiple robots. Colored circles denote corresponding robot bodies and straight lines denote robot joints. (b) An example of robot evolution parameter space after morphology matching of multiple robots. The four highlighted robots are the source and three target robots used in experiments in Section 5.1 respectively. Other semi-transparent robots are the generated intermediate robots.
etc. Suppose the kinematic- metthtt matched robots have physical parameters. Then each 0 E PR uniquely defines a new robot. Suppose the physical parameters of the source robot and the N target robots 0s E RD and f,, fr.2,..... θI,N E R respectively. On each dimension, we compute the upper
and lower bounds of the physical parameters
U MAX (105, θT, 0T,................ ..... OT,N
(2)
0 MIN (0s: 0r.1 01.
where oy and O, essentially defines the convex hull of the set of robot physical parameters in TRP that encompasses the source and all target robots. We can now use continuous function F: [0, 1] - M to define an intermediate robot by interpolation between all pairs of physical parameters
0 (1 a) 02 @ Ou
(3)
where a E [0, 1]2 is the evolution parameter that describes the normalized position robot in the convex hull. Note that by limiting a to be between O and 1, we assume that the convex hull is the set of possible intermediate robots. This a reasonable assumption since an out-of-range parameter can be physically dangerous and is also unlikely to be useful in robot continuous interpolation. The convex hull also serves as the metric space that measures the hardware difference between two robots.
In HERD (Liu et al., 2022a), the source robot and the target robot are always represented O and 1 respectively because when there are only two robots, one of them must either be the lower bound or the upper bound. Different from HERD, the source and target robots in our problem are not necessarily or 1. An example of the resulting robot evolution space and the positions of the source and target robots in the evolution space illustrated in Figure 2(b).
3.3 ONE-TO-MANY ROBOT EVOLUTION FOR POLICY TRANSFER
Suppose the source and the N target robots are represented by F(Bo) = Ms and F(B,) = Mr.....,, F(BN) My.N respectively where Br E [0, 11 Similar HERD (Liu et al., 2022a), we employ N robot evolution paths T, (F(a..1), F(a.22).... ........... F(a.,, i 1,2, ...., N where F(a,,1) F(B,) the source robot and F(a,
F(G,) the i-th target robot. Following HERD, we use K; phases of policy optimizations. At phase k. the policy is trained on policy rollouts on robots sampled from the line 0 The sampling window gradually converges to Q during training until the policy is able to achieve sufficient performance on F (( before moving on to phase 1. For all k we set the evolution step size |o, a, = g to be small enough so that each training phase is an easy sub-task.
Naively following HERD would require training through all the N evolution paths TG. However, some target robots mutually similar, at the beginning of the transfer, the robot evolution could be in roughly similar directions. Therefore, the robot evolution paths could be close to each other near the start of the paths and the policy optimization might be redundant, illustrated in Figure ((a).
4

---

Published as a conference paper at ICLR 2024
Algorithm 1 Mcta-Evolyy
Input: source robot Bo and the expert policy TP(0)) on it; target robot set B [ Bo,......... Br); on target robots;
Output: policies (TFF(S)),
(TF T((0)) (F(By) <- Meta (3:
B)
1: a + Bo, π + TF())): 4 23; // initialization
2: PMte P < Evolution Tree // temporary meta robot (htua target robot partition 2P:
3: while ||a - fsteulla 2 { do
4: T.Z 4- arg max,, [ || TBpp -
(a 1)113: optimize both path and reward I, 0,))), 1 move towards meta robot, and make sure to stay within [0, 1]
5: a <- M
6: BMtm: P _volution__eess Tree((a, B/; update Gstu and P after rohot evolution
7: if BB = then // if there is only one target robot, then the meta robot is simply the target robot
8:
return / reaching the meta robot means policy transfer completes: return the policy
9: for p in P do
10: + T Meta_Evolk catta_Evoveea, n, p); recursively transfer policy in each subtree from meta robot
11:
return II;
We propose method named Meta-Evolve that designs the N evolution paths by forcing the first mi.j Z+ intermediate robots be shared between the paths towards target robots F(3.) and F(3,) to address the redundancy issue at the start of the training. Formally, we enforce
Tk S mj Cii = j.
This means that the two paths towards F(33,) and F(3,) will first reach a shared robot F F(a F(a..m..)) before splitting their ways. In this way, both exploration and training overhead during policy transfer can be significantly saved due to path sharing. Theoretically, if all N target robots are close enough to each other, we could expect our Meta-Evolve method to yield a speedup up to O(N) compared to launching multiple one-to-one policy transfers such as HERD (Liu et al., 2022a).
As illustrated in Figure (bb, for N target robots, sharing their evolution paths essentially forms an "evolution tree" with its root node being the source robot and N leaf nodes being the N target robots. The N internal tree nodes are the intermediate robots that are last shared by the paths before the split. We name these internal tree nodes as "meta robots". Given these N - meta robots, Method-Evole Evollee method first transfers the expert policy ((B)) from the source robot 3g to the closest meta robot Meaa to obtain well-trained policy, and then recursively transfer the policy towards the target robots in each sub-tree respectively. The overall idea is illustrated in Algorithm 1.
3.4 EVOLUTION TREE DETERMINATION
Given the source and target robots, the structure of the evolution tree and the choice of meta robots significantly impacts the overall performance of the policy transfer. However, due to the huge complexity of the robots' physical parameter and its relation to the actual MDP transition dynamics in the physical world, it is extremely difficult to develop universal solution for the optimal evolution tree. We hereby propose the following heuristics for determining the evolution tree and meta robots.
Evolution Tree as Steiner Tree We aim minimize the total L travel distance in robot evolution parameter space from the source robot to all target robots. Mathematically, an undirected graph that interconnects set of points and minimizes the total [ travel distance is called the LP Steiner tree p-Steiner tree (Steiner, 1881; Gilbert & Pollak, 1968) of the point set. Then the evolution tree can be selected as the p-Steiner tree of the evolution parameter set of the source and all target robots:
min (VST, EST) (VVE))x((VEE))=.L...... E))
Σ
1 U211p
(5)
BVV (DIEE)EE
where VST and EST the vertex and edge sets of the p-Steiner tree and x(-) denotes the graph connectivity, The neighbor(s) of the source robot acts as the initial goal(s) of the evolution. If the source robot has more than neighbor in the tree, i.e. deg Ver Exr) (Bo) 1, it means the evolution paths should already be split at the source robot and the policy should be transferred in each subtree respectively. The idea is illustrated in Figure 1(b) and Algorithm 2.
Note that by using Steinee tree, we assume that the training cost of transferring the policy from robot F(O.L)) to robot F F(0g,t+1) is proportional to | 05 llp- We believe this is reasonable
5

---

Published as a conference paper at ICLR 2024
Algorithm 2 Determination of Evolution Tree and Meta Robots
Input: target robot set B {B1, f2,,.... ........'.' }; current intermediate rohot
Output: meta robot MMei: target robot partition P C 2 where UpeP B and Vp pa E P.pinp: Ø;
Gaau P + Evvouttin_TTre(xx B)
1: (Vsr, EsT) +- arg (i E))x((VV))))
((e))9))8 1|
vellp: p-Steiner tree
deg Bgr) (a) = then the current intermediate robot has only one neighbor the tree
ax <<||2 // meta robot should be the neighbor 4: P+{B); // there is no partition in the target robot set yet
3: Bstm <- arg min
5: else // the current intermediate robot has more than neighbor, so should split paths toward each subtrec
6:
BMet <- az; meta robot is the current intermediate robot itself
P+ (p' C B VV' VS. E' S EST. x((V', EE)) deg EY) ((3tB))))) = 13: partition
8: return BMtta P;
assumption since the training cost should be locally proportional to the distribution difference of the MDP transition dynamics of the two robots measured in c.g. KL divergence, and should be locally proportional to the robot hardware difference ||a;,k ai,k+illp when a a1,441llp + 0, i.e.
(F ((((,,+1)) = o(l|a,, ai.++llpp).
Implementation Details At each training phase of the policy transfer, the algorithm should aim to only reduce the expected future cost of training instead of including the past. So more optimized implementation of our method is that, at training phase k. the algorithm acts greedily to minimize the total L travel distance from the current robot ai to the meta robots BMen and then to all target robots { through the evolution tree. It can implemented by replacing source robot Go in Equation (5) with the current intermediate robot a::
arg min (VST, EST) (V))
Σ
|0 U2l1p
(6)
IV (V1,02)E
This means the evolution tree and the meta robots are temporary and are updated at the start every training phase when the robot evolution progresses, as illustrated in Algorithm 1. In practice, instead of keeping track of the entire evolution tree, we only keep the partition of the target robot set P S 2 when paths splits into subtrees and re-compute each evolution subtree after path splitting.
3.5 DISCUSSIONS
Can the Target Robots be Very Different? It possible that target robots are in opposite directions. One extreme example is that the source robot is a five-finger hand while the two target robots are ten-finger hand and two-finget gripper. Our Meta-Evolve will still be able to handle such cases correctly, but the meta robot may simply be the source robot itself. This means the evolution paths split at the start and our Meta-Evols will be reduced to multiple runs of independent one-to-one policy transfers and does not yield any speedup in performance, which reasonable. Fortunately, in practice, most commercial robots such as Sawyer, Panda and UR5e are indeed mutually similar in morphology and kinematics. So our Meta-Evolve can still be useful in these cases.
Can the Meta Robots be Learned Optimized? We envision the learning or optimization of the evolution tree and the meta robots being very challenging. Policy transfer through robot evolution relies on local optimization of the robot evolution. On the other hand, optimizing the evolution tree requires optimizing the robot evolution paths globally and needs an accurate "guess" of the future cost of policy transfer. In fact, our proposed heuristies can be viewed as using I." distance of evolution parameters to roughly guess the future policy transfer cost for constructing evolution tree. We leave the problem of finding the optimal evolution tree and meta robots as future work.
RELATED WORK
Imitation Learning across Different Robots Traditional imitation learning is designed for learning on the same robots (Ross et al., 2011; Ng al., 2000; Ho & Ermon, 2016; Duan et al., 2017). However, due to a huge mismatch in transition dynamics, these works often struggle in learning across different robots. Compared previous imitation learning methods that aim to learn across

---

Published as a conference paper at ICLR 2024
different robots directly (Radosavovic al., 2020; Liu et al., 2019; Rusu et al., 2015; Trabucco et al., 2022), we aim to employ robot evolution to gradually adapt the policy. Furthermore, our Meta-Evolve focuses one-to-many imitation where the transferred policies must work multiple target robots.
Learning Controllers for Diverse Robot Morphology Recent work has studied the problem of learning a policy/controller for diverse robots. For instance, Wang al. (2018), Huang et al. (2020) and Pathak et al. (2019) use graph neural networks to control and develop robots with different morphology that can generalize to new scenarios. Hierarchical controllers (Hejna al., 2020) and transformers (Gupta et al., 2022; Hong et al., 2021) are also shown to be effective across diverse robot morphology. contrast to these works, we do not co-develop the controller with morphology but transfer the policy from a source robot to multiple target robots.
Meta-Learning Our -eoavEvolvee is closely related the formulation of meta-learning (Finn et al., 2017; 2018; Rajeswaran et al., 2019; Nagabandi et al., 2018; Sæmundsson et al., 2018; 8:; Schoettler et al., 2020). Different from meta reinforcement learning where only the policy π is meta learned, our formulation can be viewed as the continuous update of both the policy and the transition dynamics T instantiated by setting different robot hardware parameters 0. Moreover, while meta-learning aims to learn a meta policy from scratch, in our problem, the source expert policy is given and used in policy transfer. Closely related to our approach is task interpolation for meta-learning ( (Yao et al., 2021). Different from task interpolation, our method does not require the policy work range of robots at the same time but only needs each transferred policy to work on each target robot.
Transfer Learning in RL Previous works RL transfer learning have explored transferring policies by matching certain quantities across multiple tasks. Examples include learning inter-task mappings of states and actions (Gupta et al., 2017: Konidaris & Barto, 2006; Ammar et al., 2015) and cross-task reward shaping (Ng et al., 1999; Wiewiora et al., 2003). Different from these works, we do not aim to directly find the matching between different robots but gradually evolve one robot multiple target robots through an evolution tree and transfer the expert policy along the way.
EXPERIMENTS
The design of our Meta-Evolve method is motivated by the hypothesis that by sharing the evolution paths among multiple robots through the design of the evolution tree, the overall cost of one-to-many monm policy transfer can be reduced compared to multiple one-to-one transfers. To show this, we apply our Meta-Evolve the policy transfer two types of robot learning tasks: one-to-thres policy transfer on the three manipulation tasks in Hand Manipulation Suite (HMS) (Rajeswaran et al., 201 8), and one-to-six policy transfer on an agile locomotion task in maze.
5.1 ONE-TO-THRES MANIPULATION POLICY TRANSFER
Source and Target Robots We utilize the five-finger ADROIT dexterous hand (Kumar et al., 2013) as the source robot and follow Rajeswaran et al. (2018) for the initial settings. The target robots are three robot grippers with two, three, and four fingers respectively. The target robots can be produced by gradually shrinking the fingers of the source robot, as illustrated in Figures 1 and 2(b).
Task and RL Algorithm We use the three tasks from the the task suite in Rajeswaran et al. (2018): Door, Ha r and Rel ocate illustrated in Figure 3. In Door task, the goal is turn the door handle and fully open the door; n Hammer r task, the goal is to pick up the hammer and smash the nail into the board; in Re 1 loaate task, the goal is pick up the ball and take it to the target position. We use a challenging sparse reward function where only the task completion is rewarded. We use NPG (Rajeswaran et al., 2017) as the RL algorithm in all compared methods. The source expert policy was trained by learning from human demonstrations collected from VR-empowered sensor glove.
Baselines We compare our Meta-Evolve against three baselines: (1) DAPG: we launch multiple independent direct one-to-one imitation learning using DAPG (Rajeswaran et al.. 2018); (2) HERD: we launch multiple independent one-to-one robot policy transfer with HERD; (3) Geom-Median: in this baseline, we allow only one meta robot BMts in the evolution tree. Mathematically. point that minimizes the sum of L distances to a set of points is called the L geometric median of the point set. When there is only one meta robot in the Steiner tree, the meta robot BMeu is the LP geometric median of the source and target robot evolution parameter set, i.e. BMee arg min g0 Σ ||3- (3-B1]] 11
7

---

Published as a conference paper at ICLR 2024
(a)
(b)
(d)
(c)
(1)
Figure 3: Hand Manipulation Suite (HMS) tasks (Rajeswaran al., 2018) used in our experiments: (a)
Hammer, (b) Door, and (c) Relocate. Robot Evolution paths of (d) using multiple independent HERD; (c) using geometric median as the only meta robot; and (f) our -volvevvolve
2-Hinger target mooot
3-finger target robox
4-linger robot
total
speediu
DAPG
>30K
550K
3-0KK
>-10K
#f sim
>200K
>200K
200K
>-60KK
pach
pulh 2
lotal
speedup
task
# of sim
brain
21888 2506
18612 444
1016
23796
983
64296
4357
1250
1x
Ours
path 10
11
path 12
total
speedup
(1-Steiner
#oo train
1308 245
98 17
93 25
527
=:33
19:220
2016 x: 200
2.35%
Tree)
of sim
13964
1427
964 a 97
1784 276
56228
106
816 1660:00
23156
1059
2.73x
2-finger target robut
3 finger target robot
finger target robot
total
speedup
DAPG
* train # of sim
>50K
>500
2-50
150K
>200K
>000K
2-20K
>000K Intal
path
path 2
path
speedup
HERD
# train
[0s23
6301
231 38
# of sim
57196 2401
41896 = 6629
44141
6821
143233
20062
1x
Hamner
task
(L' Geom
Ours
6295 1524
pail 4
paih 5
puth 6
230
patl
11851 3487
lotal
Spoclup
Median)
# sim
32445 = 5007
22125 + 7064
378 592
2748 369
57696 + 12549
2
Ours
path 10
II
puth 12
total
speeduu
(2-Steiner
bainn
150N
a:
a
1
214
Tree)
# of sim
18775 1479
5669 +: 394
5141 a: 940
13200 a 3355
1646 451
44431 3008
3.22x
Ours
path
12
speaduu
(1-Steiner
of train
3848 = 239
419
110
447 191
3003 = 1997
126 152
7843 = 1350
2.95%
Tree)
#f sim
22603
3007
3566 a 648
13421
1235
44333
5159
3,23x
2-llnner target robot
3-inner target robot
4-nggerr larget robot
total
spoodup
DAPG
# train of sim
>50K
-00K
>50K
150K
>-00K
>000K
>-00K
5600K
puth
poth 2
path 3
total
spoedmp
7603 1158
9850
4239
#f sim
53568 t 4287
62900 at 6640
64244
1769
180712 +: 14465
1x
Relocat
task
Ours
(L) Goom
-Median)
#f sim
Irann
39886 = 1641
9095
pulh 4
12835 a 1731
puih 5
715
1058 a P9
puth 6
3972 a 2472
549
palh
57751 w0s
lotal
2454
2.13x
3.13x
Ours
path 10
1I
path 12
total
speeduu
(2-Stcinner
of train
5717 + 2007
4833 =
1752 a 265
2198 # 18
111
16257 = 2719
1.67%
Tree)
# of sim
27006 7340
18690
+ 942 paily
7980 1544
10242 + 195
7452 + 1001
71370 + 11022
x
path
path 12
speadup
(1-Steiner
#ot train
9451 = 780
167 = 509
2: 436
967
# 206
103 13
11833 = 766
X,29%
Tree)
#
40063 3530
2343
4869 1555
7458 2940
1512
56969
3,17x
Table 1: One-to-three policy transfer experiment results Hand Manipulation Suite tasks (Rajeswaran et al., 2018). We use "mean standard deviation" from runs of five different random seeds. The details of the methods are introduced in Section 5.1. The path IDs correspond Figures ((d)
Evaluation Metrics For each compared method, the goal is to reach 80% success rate all three target robots. Due to the nature of one-to-man moc-to-many policy transfer, the total number of RL iterations or simulation epochs takes to reach this goal cannot be beforehand. So we instead report the number of policy training iterations and simulation epochs needed to reach the desired success rate.
Results and Analysis The topology of the resulting evolution tree is illustrated in Figure 3(f). As illustrated in Table 1, in terms of the total number of training iterations needed, our Meta-Evolve method is able to achieve 2.35x, 2.95x and 2.29: improvement on the three tasks respectively compared to one-to-one policy transfer using HERD. In terms of simulation epochs needed, the improvement 2.73x, x.23x and 3.17x respectively on the three tasks. Direct policy transfer with DAPG never successfully completes the task, therefore the policy was never able to be trained.
Breaking down each part of the evolution paths, we observed that in our method, the paths from source robot to the meta robots are usually the most costly and constitutes the largest portion of simulation epochs and training. The cost after splitting the path at the meta robots is smaller which is the reason for smaller total cost. The baseline of using only one meta robot in the evolution tree can also yield significant improvements, however, its performance is still inferior to using an evolution tree with multiple meta robots, which shows the general tree-structured evolution paths is necessary.
8

---

Published as a conference paper at ICLR 2024
(b)
(c)
Figure 4: Agile locomotion task in a maze. (a) Environment and task setup; (b) Evolution paths when
launching independent HERD runs; (c) Evolution paths when using L Steiner tree as evolution tree.
puth
path
puth
path
path
path
total
speedup
HERD
of train
1241 1M
2801 a
1872 2 161
2166 a 175
2812 205
3311 340
14/33 165
1x
of sim
7562
14830
9617
16970 i:
16658
19294
84931
Ours
puth 7
path
puth
pash 10
path 11
puth
Gotal
speedup
(L Steiner
of miin
1241 1 94
1666 1. 100
698 196
2009 1 332
M97 a.
428 1:83
67.39 = 721
2.20x
Tree)
of
4051
2354
34925
2
Table One-to-six policy transfer experiment results on agile locomotion task a maze. We use "mean
+ standard deviation" from runs of five different random seeds. The path IDs correspond to Figure 4(b)(c).
more interesting observation is that, for some target robots and tasks, e.g. two- and three- faree-finger target robots on Hammer task, the total cost of transferring the policy by going through multiple meta robots in the evolution tree is even smaller than cost directly transferring the policy to the target robot using HERD (Liu et al., 2022a). It shows that, transferring the policies to multiple related target through an evolution tree determined by our heuristic approach can possibly help each robot improve their own learning efficiency. We believe this phenomena deserve more future research attention.
Allaionn Studies On Hammer and Relocate e tasks, we provide ablation studies on the design choice of the distance measure used to construct the evolution tree, i.e. L vs. L distance in evolution parameter space. As illustrated in Table 1, L distance achieves better performance than L distance. A possible reason is that the hardware parameters mostly mutually independent, when empirically estimating the robot transition dynamics difference, directly adding up element-wise difference may be better than using Euclidean distance which intertwines the difference each dimension.
5.2 ONE-TO-SIX AGILE LOCOMOTION POLICY TRANSFER
Experiment Settings To show that our Meta-Evolve can generalize to diverse tasks and robot morphology, we conduct additional policy transfer experiments on an agile locomotion task illustrated in Figure 4. The goal of the robot is to move out of the maze from the starting position. The source robot is the Ant-v2 robot used in MuJoCo Gym (Brockman al., 2016). The six target robots are four-legged agile locomotion robots with different lengths of torsos, thickness of legs, and widths of hips and shoulders. The reward function is also sparse task completion reward. We use NPG (Rajeswaran et al., 2017) the RL algorithm. We report the number of training iterations and simulation epochs needed to reach 90% success rate on the task.
Results and Analysis The experiment results are illustrated in Table 2. Our Meta-Eyolve method is able to achieve 2.20x improvement in terms of the total training cost and 2 total simulation cost compared to launching multiple HERD. The improvement is less compared to manipulation policy transfer. A possible reason is that locomotion tasks are less sensitive to the morphological changes of robots than manipulation tasks, therefore benefit less from
CONCLUSION
In this paper, we introduce a new research problem of transferring an expert policy from source robot to multiple target robots. To solve this new problem, we introduce new method named Meta-Evolve that utilizes continuous robot evolution to efficiently transfer the policy through an robot evolution tree defined by the interconnection of multiple meta robots and then to each target robot. We present heuristic approach to determine the robot evolution tree. We conduct experiments on Hand Manipulation Suite tasks and an agile locomotion task and show that our Meta-Evolve can significantly outperform the one-to-one policy transfer baselines.

---

Published as a conference paper at ICLR 2024
Acknowledgment Deepak Pathak is supported in part by NSF IIS-2024594 and AFOSR FA9550-23- 1-0747.
REFERENCES
Haitham Bou Ammar, Eric Eaton, Paul Ruvolo, and Matthew E Taylor. Unsupervised cross-domain transfer in policy gradient reinforcement learning via manifold alignment. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016.
Yu-Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S Narang, Karl Van Wyk, Umar lqbal, Stan Birchfield, et al. Dexycb: A benchmark for capturing hand grasping of objects. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 9044-9053, 2021.
David Cheriton and Robert Endre Tarjan. Finding minimum spanning trees. SIAM journal on computing, 5(4):724-742, 1976.
Erwin Coumans and Yunfei Bai. Pybullet, python module for physics simulation for games, robotics and machine learning, 2016.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The spic-kitchens dataset. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 720-736, 2018
Charles Darwin. On the origin of species. Routledge, 1859.
Yan Duan, Marcin Andrychowicz. Bradly Stadie, OpenAl Jonathan Ho. Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech Zaremba. One-shot imitation learning. Advances in neural information processing systems, 30, 2017.
Marcia Fampa, Jon Lee, and Nelson Maculan. An overview of exact algorithms for the euclidean steiner tree problem in n-space. International Transactions in Operational Research, 23(5): 861-874, 2016.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126-1135. PMLR, 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning Advances in neural information processing systems, 31, 2018.
Edgar N Gilbert and Henry O Pollak. Steiner minimal trees. SIAM Journal on Applied Mathematics, 16(1):1-29, 1968.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18995-19012, 2022.
Abhishek Gupta, Coline Devin, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Learning invariant feature spaces transfer skills with reinforcement learning. arXiv preprint arXiv:1703.029.. 2017.
Agrim Gupta, Linxi Fan, Surya Ganguli, and Li Fei-Fei. Metamorph: Learning universal controllers with transformers. arXiv preprint arXiv:2203. 2022.
Donald Hejna, Lemrel Pinto, and Pieter Abbeel. Hierarchically decoupled imitation for morphological transfer. In International Conference on Machine Learning, pp. 4159-4171. PMLR, 2020.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural
information processing systems, 29:4565-4573 2016.
10

---

Published as a conference paper at ICLR 2024
Sunghoon Hong, Deunsol Yoon, and Kee-Eung Kim. Structure-awant transformer policy for inhomo- geneous multi-task reinforcement learning. International Conference on Learning Representa- tions, 2021.
Wenlong Huang, Igor Mordatch, and Deepak Pathak. One policy control them all: Shared modular policies for agent-agnostic control. In International Conference on Machine Learning, pp. 4455 4464. PMLR, 2020.
Oussama Khatib. unified approach for motion and force control of robot manipulators: The operational space formulation. IEEE Journal on Robotics and Automation, 3(1):43-555 1987.
Pradeep K Khosla and Takeo Kanade. Parameter identification of robot dynamics. In 1985 24th IEEE conference on decision and control, pp. 1754-1760. IEEE, 1985.
George Konidaris and Andrew Barto. Autonomous shaping: Knowledge transfer in reinforcement learning. In Proceedings of the 23rd international conference on Machine learning, pp. 489-496, 2006.
Vikash Kumar, Zhe Xu, and Emanuel Todorov. Fast, strong and compliant pneumatic actuation for dexterous tendon-driven hands. In 2013 IEEE international conference on robotics and automation, pp. 1512-1519. IERE, 2013.
Fangchen Liu, Zhan Ling, Tongzhou Mu, and Hao Su. State alignment-based imitation learning. arXiv preprint arXiv: 1911. 10944, 2019.
Xingyu Liu, Deepak Pathak, and Kris M. Kitani. HERD: Continuous Human-ta toan--tt-oooott Evolution for Learning from Human Demonstration. In The Conference Robot Learning (CoRL), 2022a.
Xingyu Liu, Deepak Pathak, and Kris M. Kitani. REvolveR: Continuous Exolutionary Models for Robot-to-robot Policy Transfer. In The International Conference on Machine Learning (ICML), 2022b.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108. 10470, 2021.
Mayank Mittal, Calvin Yu, Qinxi Yu, Jingzhou Liu, Nikita Rudin, David Hoeller, Jia Lin Yuan, Ritvik Singh, Yunrong Guo, Hammad Mazhar, et al. Orbit: A unified simulation framework for interactive robot learning environments. IEEE Robotics and Automation Letters, 2023.
Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning adapt dynamic, real-world environments through meta-reinforcemmet learning, arXiv preprint arXiv: 1803. 1347, 2018.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application reward shaping. Ieml, volume 99, pp. 278-287, 1999.
Andrew Y Ng. Stuart J Russell, et al. Algorithms for inverse reinforcement learning. Icml, volume 1, pp. 2, 2000.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Deepak Pathak, Chris Lu, Trevor Darrell, Phillip Isola, and Alexei A Efros. Learning to control self-assembling morphologies: a study of generalization via modularity. NeurlPS, 2019.
Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference robotics and automation (ICRA), pp. 3803-3810. IEEE, 2018.
Ilija Radosavovic, Xiaolong Wang, Lerrel Pinto, and Jitendra Malik. State-only imitation learning for dexterous manipulation. arXiv preprint arXiv:2004.04653 2020.

---

Published as a conference paper at ICLR 2024
Aravind Rajeswaran, Kendall Lowrey, Emanuel Todorov, and Sham Kakade. Towards generalization 2017.
and simplicity in continuous control. arXiv preprint
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. In Proceedings of Robotics: Science and Systems (RSS), Pittsburgh, Pennsylvania, June 2018. doi: 10.15607/RSS.200 /22/// 215607/RRSS.208.XXVV.. X.XI.099
Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. Advances in neural information processing systems, 32, 2019.
Gabriel Robins and Alexander Zelikovsky. Improved steiner tree approximation in graphs. In SODA, pp. 770-779, 2000.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings,
2011.
Andrei Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy 2015.
distillation. arXiv preprint arXiv:/511.0663 2015.
Steindór Sæmundsson, Katja Hofmann, and Mare Peter Deisenroth. Meta reinforcement learning with latent variable gaussian processes. arXiv preprint arXiv: 07.50055115 2018.
Gerrit Schoettler, Ashvin Nair, Juan Aparicio Ojea, Sergey Levine, and Eugen Solowjow. Meta- reinforcement learning for robotic industrial insertion tasks. In 2020 IEEE/RSI International Conference on Intelligent Robots and Systems (IROS), pp- 9728-9735. IEEE, 2020.
Naveed A Sherwani. Algorithus for VLSI physical design automation. Springer Science & Business Media, 2012.
Warren D Smith. How to find steiner minimal trees in euclidean d-space. Algorithmica, 7:137-177, 1992.
Jakob Steiner. Gesammelte Werke: herausgegeben auf Veranlassung der Königlich Preussischen Akademie der Wissenschaften, volume 1. G. Reimer, 1881
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference intelligent robots and systems (IROS), pp. 23-30. IEEE, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Brandon Trabucco, Mariano Phielipp, and Glen Berseth. Anymorph: Learning transferable polices by inferring agent morphology. In International Conference on Machine Learning, pp. 21677-21691. PMLR, 2022.
Tingwu Wang. Renjie Liao, Jimmy Ba, and Sanja Fidler. Nervenet: Learning structured policy with graph neural networks. ICLR, 2018
Eric Wiewiora, Garrison W Cottrell, and Charles Elkan. Principled methods for advising reinforce- ment learning agents. In Proceedings of the 20th international conference on machine learning (ICML-03), pp. 792-799, 2003.
Huaxiu Yao, Linjun Zhang, and Chelsea Finn. Meta-learning with fewer tasks through task interpola- tion. In International Conference on Learning Representations, 2021.
Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martin-Martin, Abhishek Joshi, Soroush Nasiriany, and Yifeng Zhu. robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint 20X::20009.1229 2020.
12

---

Published as a conference paper at ICLR 2024
(n)
Figure 5: The evolution tree from the source robot, i.e. (a) ADROIT five-finger hand, to three target real commercial robots: (b) Jaco robot with three-finger Jaco gripper, (c) Kinova3 robot with two-finger Robotiq-85 gripper, and (d) IIA robot with two-finger Robotiq- 140 gripper.
(b)
(c)
(d)
(a)
(e)
(f)
(h)
Figure 6: (a) Robot Evolution paths of using multiple independent HERD (paths 1, and 3) and our Meta- Evolve (paths 4, 5, 6, and 8); (b)-(g) Visualization of the trained policy rollouts on the object manipulation task. The green semi-transparent object denotes the goal position; (h) Real-world deployment of the target robot policy real Kinova3 machine.
path 11313 + 864
path
path
total
speedup 1
HERD
train
13405 985
13719 = 1077
38437 + 2883
sim
52358
65340
68127
185825
Ours
path 4
path 5
path
path
path 8
total
speedup
(1-Steiner urin
10810 = 1366
4364 1609
3291
1443
761 #
855 # 396
20081 4509
1.91x
Tree)
sim
48398
20352
17861
1569
1744
89924
2.07
Table One-to-three policy transfer experiment results on DexYCB manipulation task. We use "mean standard deviation" from runs of five different random seeds. The path IDs correspond to Figure 6(a).
A ADDITIONAL EXPERIMENTS ON REAL COMMERCIAL ROBOTS
To show that our Meta-Evolve can be applied to real robots and real-world tasks, we conduct an additional set of experiments transferring an object manipulation policy to multiple real commercial robots.
Source and Target Robots The source robot is the same ADROIT hand robot (Kumar et al., 2013) used in Section 5.1. The three target robots follows and illustrated in Figure ((b)
Jaco: Jaco a 7-DoF robot produced by Kinova Robotics.
equipped with the Jaco Three-Finger Gripper, three-finger gripper with multi-jointed fingers.
Kinova3: Kinova3 is 7-DoF robot produced by Kinova Robotics. equipped with the Robotiq-85 Gripper, the 85mm variation of Robotiq's 's multi-purpose two-finger gripper
13

---

Published as a conference paper at ICLR 2024
IIWA: IIWA is an industrial-gres -rradd 7-DoF robot produced by KUKA. is equipped with the Robotiq-140 Gripper, the 140mm variation of Robotiq's 's multi-purpose two-finger gripper.
We follow the high-fidelity robot arm models introduced in Zhu et al. (2020) for the detailed physical specifications of the target robots to minimize the sim-to-real gap. Note that in the simulation model of the source ADROIT hand, the robot is rootless, i.e. connected to a virtual mount base via translation and rotation joints. We adopt the same procedure introduced in Liu et al. (2022a) to attach the ADROIT virtual mount base the end-effector of the 7-DoF robot arm. The end-effector of the robot arm is controlled by an Operational Space Controller (OSC) (Khatib, 1987) that moves the end-effector its desired 6D pose with PD control schema. We follow Zhu al. (2020) for the implementation of OSC. Apart from the states of the ADROIT hand, the 6D pose of the robot end-effector is additionally included in the state of the robot. During robot evolution, the original ADROIT arm shrinks and the five-finger ADROIT hand gradually changes to be the target gripper. Please refer to Liu et (2022a) for more details on the idea behind this implementation. The resultant L evolution tree is illustrated in Figure 5 and is used in our experiments.
Task and RL Algorithms The task setup is illustrated in Figure 6. The goal of the robot is to pick up the object and take it to the desired goal position. The task is considered success if the distance from the object to the goal is sufficiently small. The reward function is sparse task completion reward. We use NPG (Rajeswaran et al., 2017) as the RL algorithm. The source expert policy is trained by learning from the human hand demonstrations in Dex YCB dataset (Chao et al., 2021). We report the number of training iterations and simulation epochs needed to reach 80% success rate on the task.
Results and Analysis The experiment results are illustrated in Table 3. Our Meta-Evolve method is able to achieve 1.91 improvement in terms of the total training cost and 2.07x total simulation cost compared to launching multiple HERD. This shows that our Meta-Evolve can be applied to real commercial robots. Moreover, to show that the transferred policy can be used on real target robots, we conduct real-world experiment and deploy the target robot policies on Kinova3 on the corresponding real machine illustrated in Figure 6(h). Please refer to the our project website for more details.
B ADDITIONAL DISCUSSIONS
Can the physical parameters of all robots be known? It is easy to obtain all necessary physical parameters of a robot. The ultimate goal of our method is train a policy that can be deployed on real robots. In order to do such real-world experiments, we need to obtain the robot physically. At that time, we will have access to every specification of the robot:
If we purchase, borrow or rent a commercial robot: When selling their robots, licensed robot manufacturing companies would release detailed parameters of their robots. Besides, the controller software such as Operational Space Controller (OSC) (Khatib, 1987) is usually also released for the robot by the manufacturers. These controller software can only work correctly when all the necessary physical parameters of the robots are matched with the actual hardware, including the inertia and mass of every robot body, and damping and gain of every motor etc. Users with sufficient robotics expertise can easily infer the accurate physical parameters from the released control software or the manual of the robot provided to the users.
If we create our own robot: Nowadays, the mechanical components of new robots manufactured by printing 3D CAD models. Therefore, all physical parameters of the mechanical parts of the robot can be easily calculated using the 3D CAD design software. This is also how the robot manufacturing companies obtain the parameters for their own commercial robots. If the robot we obtained has missing or unknown parameters: It not recommended to use robot with missing or unknown parameters because it might be dangerous to do so. In the rare and extreme case where we are forced to use a robot with missing or unknown parameters, the methods for accurately measuring the parameters of unknown real robots were already developed in the 1980s (Khosla & Kanade, 1985) and have been maturally used in the robotics industry for decades since then.
If we attach external components to the robot: External components may introduce additional physical parameters such as friction coefficient of the auxiliary gripper finger parts etc. These external parameters can be easily and accurately measured in lab experiments. On the other hand, it is not recommended use an external components without knowing its detailed parameters
14

---

Published as a conference paper at ICLR 2024
because it might be dangerous do so. Please refer to related literature on mechanical or materials engineering for more details on how to measure these parameters in the lab.
Real-robot Simulation Experiments? If we want to take advantage of the power of deep reinforcement learning in solving robotics tasks, it is imperative to collect sufficient amount of data, and performing large-scale simulation is the most convenient way to do so. On the other hand, modern simulation engines such as MuJoCo (Todorov et al., 2012), Pybullet (Coumans & Bai, 2016) and Isaac (Makoviychuk et al., 2021) allows highly accurate physics simulation to be performed. Futhermore, simulation frameworks built upon these engines such as robosuite (Zhu et al., 2020) and Orbit (Mittal al., 2023) can simulate robot actions and robot-object interactions with very high fidelity. It is true that as long as simulation is used, there is always Sim-to Sim-to-real gap. Fortunately. there has been numerous works on reducing the Sim-to-real gap in robotic control (Peng et al., 2018; Tobin et al., 2017). Sim-to-real transfer is not the focus of our work, therefore is not discussed in depth in our paper.
Scalliggtransformatioos on the evolution parameters a and its effect on Meta-Evolve? There could be many ways to transform the physical parameter Oş and OT, to be evolution parameters 33 by Bo f(0s) and B, = f(Or,)) .) In the main paper, we present a simple and intuitive way by normalizing each element of 0 to [0. 1] through linear transformation. If 7.2 distance is used to construct the evolution Steiner tree, the meta robots depend on the choice of the transformation function f. However, if I distance is used instead, the meta robots will not depend on the choice of f. as long as f is monotonic on every input dimension. As introduced in Equation (8), the L distance of two vectors is the sum of element-wise absolute difference of the two vectors. Given the current intermediate robot a E R D and target robots 31, . E IR DR, the first meta robot in the Steiner tree to reach from is given by element-wite clamp operation to the convex hull boundaries of the target robots:
BMesa MMAX(B,,MIN(C Br))
where Bu = M) B2..... BN1)
and 32 = 12...,, }} are the element-wise upper and lower bounds of the target robot evolution parameter convex hull. This is because the convex hull defines the spanning range of the target robot parameters and should be used as the indicator of where to split the evolution path. As long as f is monotonic on every dimension of the parameter, the meta robot given by Equation (7) is invariant to f.
Can Meta-Evolve Evolve generalize to any known robot configurations? Our Meta-Evoly can gener- alize to any known robot configurations. For any source robot and N target robots, we can always use the method described in Section 3.2 to match the robot kinematic trees and state/action spaces. Whether the policy transfer can work on that set of source and target robots depends the task as well as the policy. For example, we do not expect a manipulation policy on a five-finger gripper can be transferred some four-legged agile locomotion robots, though can still define the intermediate robots between them. As long as the task and robot settings are reasonable, c.g. transferring a manipulation policy from one robot gripper some other robot grippers, our Meta-Evolve should be able to deal with these cases, as shown by the experiments in our paper.
Can Meta-Evolve generalize unknown robot configurations? Our Meta-Evole does require knowing the configurations of the source and target robots. This is a reasonable assumption because we only deal with predetermined source and target robots. When we deploy the trained policies on the robots in the real world, we would need to first obtain these robots physically. Then at that time we will have access to everything about the robots. Please refer to the discussions on how to obtain the physical parameters of a robot, i.c. the first paragraph of Section B, for more details.
C INTRODUCTION TO STEINER TREE
In RD, the Steiner tree problem is to find a network of minimum length interconnecting a set B of N given points. Such networks can be represented by a tree. The set of the tree nodes can consist of the points in B, known as terminals, and possibly of additional points, known as Steiner points. The length of the network is defined as the sum of the lengths of all the edges in the tree. Without allowing Steiner points, the problem is reduced to the well-known minimum spanning tree problem (Cheriton & Tarjan, 1976). Allowing additional Steiner points can possibly reduce the length of the network, but can also make the problem harder.
15

---

Published as a conference paper at ICLR 2024
A
A
B
B
S
C
(a)
(b)
(c)
(d)
Figure 7: (a) An L' Steiner tree of three points A, B and C' consists of additional two Steiner points Sj and S2; (b) Another solution to L Steiner tree of the same three points A, B and C consists of two different Steiner points S1 and Sa; (c) The L Steiner tree of three points A. and C consists of one Steiner point S1 when all angles of A AC' are all smaller than 120%; (d) The L Steiner tree of three points A, B and D consists no Steiner point and reduced an L minimum-spaaning tree when one of angles of AABD larger than 120"
The Steiner trees can be classified based on the metric used to measure the edge lengths. When the edge length is measured with T." norm, the Steiner tree is known as p-Steiner tree. In the main paper, we did ablation studies both L' and L norms for constructing the Steiner tree. In this section, provide more detailed background on 1-Steiner tree and 2-Steiner tree.
C.I 1- I-STEINER TREE
The 1. distance of two points a [01, .gy a RD and 3= [11 :2............ 3pl RD, also known as Manhattan distance, is the sum of element-wise absolute difference and is defined as
D ||a -111 = Σ 0: - -
(8)
By using I distance of evolution parameters of two robots to measure their MDP difference as the guideline for constructing Steiner tree, we not only assume the changes of each robot hardware pa- rameter contribute equally to the change of MDP, but also assume their effect on MDP is independent and can be summed up directly. Without additional information on the actual robot hardware and task, we believe this is reasonable assumption.
1-Steiner tree of a point set B. also known as Rectilinear Steiner tree, is the undirected graph that interconnects B and minimizes the total L lengths of its edges. Finding the L' Steiner tree one of the core problems in the physical design of electronic design automation (HDA). In VLSII circuits, wire routing is only carried out by metal wires running in either vertical or horizontal directions (Sherwani, 2012).
L Steiner tree problem known be an NP-hard problem. However, multiple approximate and heuristic algorithms have been introduced and used in VLSI design. Using algorithms introduced by Robins& Zelikovsky (2000), for N terminal points, a good approximate solution L' Steiner tree can be found in O(N log N) time. The solution to L' Steiner tree of specific set of points may not be unique. Examples of L Steiner tree are illustrated Figure 7(a)(b).
C.2 2-STEINER TREE
The 2 distance of two points [ai, Q2 a.g...., apl R and B [[1, ,,...., 3pp R., also known as Euclidean distance, is the square root of element- summation the difference squares and is defined
D ||0 1l-B612- = Σ (a: - -3)2
1=1
A 2-Steiner tree of a point set B, also known as Euclidean Steiner tree, is the undirected graph that interconnects B and minimizes the total 2 lengths of its edges. L distance of evolution parameters of two robots intertwines the effect of the difference each dimension. We believe this is one of the reasons that 2 Steiner tree as the evolution tree shows worse performance in one-to one-to-many policy transfer than I Steiner tree.
16

---

Published as a conference paper at ICLR 2024
In L Steiner tree, terminal has degree between 1 and 3 and Steiner point has degree of 3. An L Steiner tree of N terminals have at most - Steiner points and all Steiner points must lie in the convex hull of the terminals. The Steiner point and its three neighbors in the tree must lie in plane, and the angles between the edges connecting the Steiner point to its neighbors are all 120".
I Steiner tree problem is also known to be an NP-hard problem. Similar I.* Steiner tree, multiple approximate and heuristic algorithms have been introduced for I,² Steiner tree (Fampa et al., 2016) where good solution can be found in O(N log N) time. The Euclidean Steiner tree of three vertices of a triangle is also known as the Fermal point of the triangle, illustrated in Figure 7(c)(d). The solution to L Steiner tree of specific set of points may not be unique.
C.3 OUR IMPLEMENTATION
Though both L' and 2 Steiner tree problems are NP-hard, fortunately, there exist multiple heuristic algorithms for approximate solutions in O(N log N) time for N target robots. We used Robins & Zelikovsky (2000) for computing L Steiner tree and Smith (1992) for computing L,2 Steiner tree in our implementation of finding evolution trees. practice, we not expect to deal with an extremely huge number of target robots. We expect the number of robots being dealt with to be under 20, which means the CPU time spent to compute both L and L Steiner trees is negligible.
D ROBOT EVOLUTION SPECIFICS
For robots with different state and action spaces, Meta-Evolve converts different state and action spaces into the same state and action space with different transition dynamics. Specifically, the kinematic trees of all robots are unified by adding additional bodies and joints, though the new bodies and joints may be zero in their physical parameters, e.g. zero mass, zero sizes, zero motors etc, so that the original expert policy remains intact. The zeros are inserted the correct positions of the state vectors of different robots to map them to the same state space. In this section, we provide more details on the evolution of the robots used in our experiments.
Manipulation Policy Transfer Experiments We illustrate the kinematic tree of the source ADROIT Figure 8. During
robot Kumar et al. (2013) used in our manipulation policy transfer experiments evolution, all revolute joints gradually freeze to have motion range of O. On the other hand, the prismatic joints are initially frozen with a motion range O, and some of their ranges gradually increase until the same full range.
During robot evolution, the body of the ring finger gradually shrinks to zero-size and disappears for all target robots. Besides, other fingers may also gradually shrink and disappear for certain target robots, e.g. the middle finger and the little finger will shrink and disappear for the two-finger target robot. Our evolution solution includes the changing of D 65 independent robot parameters resulting in an evolution parameter space of [0, 1]66.
Agile Locomotion Policy Transfer Experiments During robot evolution of the agile locomotion policy transfer experiments, the lengths of the body, thickness of the legs, and the widths of the shoulder and hip change. This is implemented by changing the sizes of the torso frame, the size of legs as well as leg mounting positions. The solution includes the changing of D = 5 independent robot parameters resulting in an evolution parameter space of [0, 1]5.
E TRAINING DETAILS
Hyperparameter Selection. We present the hyperparameters of our robot evolution and policy optimization Table 4. To fairly compare against HERD (Liu et al., 2022a), the two methods should be compared under their respective optimal performance. Fortunately, our Meta-Evolve and HERD share the same set of hyperparameters while achieving their own optimal performance. This is discovered by searching the optimal combinations of hyperparameters both methods. Another baseline method DAPG (Rajeswaran et al., 2018) uses the same RL hyperparameters illustrated in Table 4.
Performance Threshold for Moving to the Next Intermediate Robot.
We used success rate as the indicator for deciding whether to move to the next intermediate robot during policy transfer.
17

---

Published as a conference paper at ICLR 2024
Joint
Joint
Revolute Joint
Joints Frozen in Source
Target Robot
Rohot
Hand, Evolve be Frozen
Target Rohot
Body
Figure 8: Kinematic tree of dexterous hand robot. All revolute and free joints will gradually freeze during evolution. The prismatic joints initially frozen and evolve to be active.
Hyperparameter
Value
RL Discount Factor
0.995
GAE
0.97
NPG Step Size
0.0001
Policy Network Hidden Layer Sizes
(32,32)
Value Network Hidden Layer Sizes
(32,32)
Simulation Epoch Length
200
RL Traning Batch Size
12
Evolution Progression Step Size g
0.03
Number of Sampled Evolution Parameter Vectors for Jacobian Estimation in HERD Runs
72
Evolution Direction Weighting Factor À Sample Range Shrink Ratio
1.0
0.995
Success Rate Threshold for Moving the Next Training Phase
66.7%
Table The value of hyperparameters used in our experiments.
Specifically, the policy transfer moves to the next intermediate robot if and only if the success rate on the current intermediate robot exceeded a certain threshold. As illustrated in Table 4, 66. 66.7% is roughly the best option for both HERD and Meta-Evols Using higher success rate threshold may waste training overhead on intermediate robots and slow down policy transfer, while using a lower success threshold may sacrifice sample efficiency in later stages of policy transfer due sparse-reward settings.
Evaluation Metrics. We followed HERD (Liu et al., 2022a) and adopted the training overhead needed to reach 80% success rate as our evaluation metric. The core idea of using this evaluation metric is compare the efficiency of the policy transfer when the difficulty of transferring each policy is unknown beforehand. Using alternative success rate higher than 80% as the evaluation metric could also feasible. However, since the success rate threshold for moving on to the next intermediate robot is 66.7%, the success rate increase from 80% to higher one can only happen after the policy transfer completed. Therefore, the remaining part of training for reaching a higher success rate is simply vanilla reinforcement learning on the target robots and is irrelevant to our problem of inter-robot policy transfer, so should not be included in the evaluation.
Training Platforms. We use PoTorch (Paszke al., 2019) (Rajeswaran et al., 2017) as the RL algorithm all manipulation policy transfer and agile locomotion transfer experiments. We used MuJoCo (Todorov et al., 2012) the physics simulation engine.
deep learning framework and NPG
F VISUALIZATIONS
On the three Hand Manipulation Suite (Rajeswaran et al., 2018) tasks, we provide visualizations for the expert policy on the source robot and the transferred policies on three target robots Figure 9. As shown in the visualizations, during policy transfers, the original behaviors can be generally
18

---

Published as a conference paper at ICLR 2024
(a)
(b)
(c)
(d)
Figure 9: Visualization of the trained policy rollouts on Hand Manipulation tasks Suite (Rajeswaran al., 2018). From the first to the third row: Hammer task, Door task and Relocate task. From left to right: (a) source robot; (b) 2-finger target robot; (c) 3-finger target robot; (d) 4-finger target robot.
maintained in the original expert policy and also transfer it to each target robot. Please refer the attached supplementary video or our project website for more details on the visualizations.
19


Reviews:
[{"review_id": "C30nqSjTWv", "reviewer": "Reviewer_6cdy", "summary": "The paper presents a novel method for transfer learning of policies from one source robot to multiple target robots. In other to achieve this, the authors extend HERD (and REvolveR) with a novel robot kinematic evolution tree that is based on Steiner Trees. The experiments showcase that the proposed method performs better than HERD and other baselines in several experiments.", "questions": "1) How can the proposed method handle significant dynamics differences between the source and target robots? Can it even do that?\n2) How does the proposed method fit inside the Sim2Real literature?\n3) How can the proposed method integrate model-based RL/learning? Why did the authors not experiment with this?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The idea is using p-Steiner trees to represent the robot kinematic evolution tree is interesting, intuitive, simple and seems effective.\n- The method clearly accelerates transfer learning compared to the baselines provided.\n- The paper is generally well-written and the main messages are effectively conveyed.", "weaknesses": "- My main concern is the practicality of the proposed method. In other words, how can this be used in real(ish) world applications? I explain further:\n    - First, as the name suggests the method explores in the kinematic space of the robots. What happens if the dynamics differ drastically? An example would be, having a robot that has the same kinematic structure but twice the masses. Can the method handle this? It seems that no. Isn't changing the dynamics but keeping the kinematics the same a different robot? I think this deserves more intuition and explanation.\n    - Then, I am not sure how we should interpret the additional experiments on commercial robots. I cannot see the added value of those experiments and how they contribute towards convincing us that the method is applicable to real-world situations/robots. The robots are purely position-controlled and the behaviors are quite simple.\n    - Lastly, the comments on Sim2Real are weak imho. I do not see why the paper is not relevant to Sim2Real methods. The paper/method claims one to many policy transfer with *different dynamics* involved (per robot).\n- The authors have chosen to consider a model-free approach to RL/transfer learning, while obviously the models are known (at least the kinematics part). This choice should be better motivated and highlighted in the text.\n- **One of the videos is not properly anonymized; we can clearly see the face of someone performing the experiments.** Is this part of the video part of the DexYCB dataset? If this is the case, it should have been highlighted. If not (and the one performing the experiments is one of the authors), the AC should take a position on this as I am not sure if this is allowed by ICLR regulations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jbHXXNkDyF", "reviewer": "Reviewer_NXFw", "summary": "In Meta-Evolve: Continuous Robot Evolution for One-to-many Policy Transfer, the authors introduce a novel method of policy transfer which they claim is able to improve the efficiency of policy transfer between robots when multiple recipient bodies are involved, by performing policy transfer along an “evolutionary” line of intermediary bodies. Using “kinematic tree matching” and some geometric heuristics (based on Steiner trees), they are able to identify bodies which can be used in the paths of multiple recipients, reducing the total number of bodies where training is needed to enable the policy transfer. They demonstrate significant increases in efficiency over alternative methods which do not use such intermediates.", "questions": "1. How common is the problem addressed by this paper, of transferring an effective policy from one robot morphology to multiple others at once?\n2. Why was the only the 1-Steiner tree attempted in the door task?\n3. Why were so few seeds used for the results?\n4. Why was the value of 80% selected?\n5. Did the authors consider the relationship between the reduction in the total length of the paths in the graph (as compared to independent paths) and the reduction in the training time? Were these correlated?\n6. Why wasn’t DAPG run to completion? How effectively was it able to perform the task, given that it did not complete in the time provided?\n7. Do the authors have any comments on why (e.g. in the Hammer task, on path 8-10-11) their outperformed HERD on one-to-one transfer?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is very well written with nice graphical explanations of the technique and clear and well reasoned theoretical work. The authors were careful to make sure that the technical details were written unambiguously and the descriptions of the underlying mathematics were excellent. The results of the work seem compelling and are well described.", "weaknesses": "It is not obvious to me, as someone who does not deal much with the realm of physical robots, how common the problem addressed in this paper is—my uninformed guess would be that it is not so common, but the technique does not suffer much for this. It is not obvious that the work has much to do with evolution, in either the biological or computational senses, except in its use of trees which are reminiscent of the tree of life. It is also not obvious why the word “meta” was chosen, especially given that it has other connotations in Reinforcement Learning. It is not established within the paper how efficacious the kinematic tree matching methods are—although it seems to be good enough at transferring, it would be interesting to see how this matching deals with adverse cases. The number of training runs in the experiments was quite small, with only 5 random seeds. It would have been better to see a higher-quality evaluation of the method, but the presented results seem clear. It is also not obvious why some of the thresholds in the evaluation were selected, and it would be interesting to see if a change from 80% to 70% or 90% would change the order of the methods. I suspect that it would not, but the choice of selecting a single threshold leaves the question open.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lVDj2QX590", "reviewer": "Reviewer_e334", "summary": "The paper discusses the Meta-Evolve method, which aims to transfer expert policies from a source robot to multiple target robots. The paper introduces a new research problem and proposes a solution that utilizes continuous robot evolution and a robot evolution tree. The experiments conducted on hand manipulation tasks and agile locomotion tasks show that Meta-Evolve outperforms one-to-one policy transfer baselines. Ablation studies and discussions on handling different target robots and learning or optimizing the evolution tree are also presented. Overall, the paper highlights the effectiveness of Meta-Evolve in inter-robot policy transfer and suggests areas for future research.", "questions": "Can and how your method transfer to the real world robotic problems? Like manipulation tasks.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper highlights the effectiveness of Meta-Evolve in inter-robot policy transfer, as demonstrated through experiments on hand manipulation tasks and agile locomotion tasks (very interesting). The results show that Meta-Evolve outperforms one-to-one policy transfer baselines in terms of training and simulation costs.", "weaknesses": "The experiments are mainly limited to hand manipulation tasks, which can be easily represented by tree structures. Does your main idea still works on modular robots (like 3d voxel-based robot)?\n\nReference: Nick Cheney, Robert MacCurdy, Jeff Clune, and Hod Lipson. Unshackling evolution: evolving soft robots with multiple materials and a powerful generative encoding. In GECCO ’13, 2013.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "50HYVIZ5ys", "reviewer": "Reviewer_Escv", "summary": "This paper introduces a method to perform expert policy transfer from one robot to many other robots of different morphology. As opposed to previous works, this method defines a tree structure that finds a meta-robot to act as an intermediary fine-tuning platform before the policy is transferred to a target robot (i.e. finding a common robot-morphology ancestor in the robot evolution tree). This robot evolution tree allows for faster transfer between source and target robots because the policy only needs to be transferred to the meta robot once, as opposed to N times (as with previous methods). The paper experiments on several transfer tasks, including manipulation tasks where the morphology of the robotic hand changes from source to target.", "questions": "1. How does changing the morphplogy of a robot affect the optimal policy?\n2. How are the authors training the new policies on the new robots?\n3. Does the simple fact that the authors are fine-tuning the same policy more times on the baselines lead to poor performance?\n4. Can you please describe quantitatively/qualitatively the quality of the learned target policies between the different methods?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is clearly written and motivates the problem well. This method exposes clear gaps in the literature regarding and proposes a simple and novel method that can improve the sample efficiency of existing robot-robot transfer methods. Additionally, the authors propose an interpretable way to measure the distance between robot hardware morphologies in equation (3). \n\nMainly, the authors' proposal of an ancestral meta-robot common to all target robots is an interesting idea worth exploring. It is clear from the results reported in the paper that this idea speeds up training time/simulation time.", "weaknesses": "There are several weakness in the paper that I think need to be addressed.\n\n1. \n-  In section 3.2, the authors discuss kinematic tree matching between robots with similar kinematic structures. However, I am concerned that they are claiming that just because two robots share a similar kinematic tree, that they also share a similar expert policy. In my understanding, the kinematic tree matching would relate a pair of robots with 2 fingers and 3 fingers more closely than a pair robot with 2 fingers and 4 fingers. However, it very well may be the case that a robot with 2 fingers will act more similarly to a robot with 4 fingers than one with 3 fingers.\n- Further, the convex hull of robot hardwares might be irrelevant if the robots close in hardware-space are far in policy space\n\n2. I am curious about the method the authors use to actually train the policies on the new robots. The description of generating new robots is clear but the method for actually transferring the policies is not.\n\n3. I am concerned about the notion of neural plasticity in this problem. Simply fine-tuning the policy more times might lead to the degredation in performance that we see in the HERD and REvolveR methods\n\n4. The authors do not demonstrate the quality of the new policies. I know they are training to achieve some specified success rate but I think more experimentation on what the transferred policies are is important to discuss in this paper", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)