One Policy to Run Them All: an End-to-end Learning Approach to Multi-Embodiment Locomotion
Authors: Nico Bohlinger
Grzegorz Czechmanowski
Maciej Piotr Krupka
Piotr Kicki
Krzysztof Walas
Jan Peters
Davide Tateo
Venue: CoRL 2024 (2024)
Keywords: Locomotion
Reinforcement Learning
Multi-embodiment Learning
TL;DR: We propose a neural network architecture that can learn locomotion over multiple legged robot embodiments and morphologies.

Abstract: Deep Reinforcement Learning techniques are achieving state-of-the-art results in robust legged locomotion.
While there exists a wide variety of legged platforms such as quadruped, humanoids, and hexapods, the field is still missing a single learning framework that can control all these different embodiments easily and effectively and possibly transfer, zero or few-shot, to unseen robot embodiments.
To close this gap, we introduce URMA, the Unified Robot Morphology Architecture. Our framework brings the end-to-end Multi-Task Reinforcement Learning approach to the realm of legged robots, enabling the learned policy to control any type of robot morphology.
The key idea of our method is to allow the network to learn an abstract locomotion controller that can be seamlessly shared between embodiments thanks to our morphology-agnostic encoders and decoders. This flexible architecture can be seen as a first step in building a foundation model for legged robot locomotion.
Our experiments show that URMA can learn a locomotion policy on multiple embodiments that can be easily transferred to unseen robot platforms in simulation and the real world.

One Policy to Run Them All: an End-to-end Learning Approach to Multi-Embodiment Locomotion
Nico Bohlinger', Grzegorz Czechmanowski Maciej Krupka', Piotr Kicki
Krzysztof Walas Jan Petersl.3,5g Davide Tateo Department of Computer Science, Technical University of Darmstadt, Germany Institute of Robotics and Machine Intelligence, Pozman University of Technology, Poland
3 German Research Center for Al (DFKI), Research Department: Systems AI for Robot Learning Hessian.AI; "Centre for Cognitive Science
IDEAS NCBR, Warsaw, Poland:
Figure 1: Top We train a single locomotion policy for multiple robot embodiments in simulation. Bottom We can transfer and deploy the policy on three real-world platforms by randomizing the embodiments and environment dynamics during training.
Abstract:
Deep Reinforcement Learning techniques are achieving state-of-the the-artt results in robust legged locomotion. While there exists wide variety of legged platforms such as quadruped, humanoids, and hexapods, the field is still missing a single learning framework that can control all these different embodiments easily and effectively and possibly transfer, zero or few-shot, to unseen robot embod- iments. We introduce URMA, the Unified Robot Morphology Architecture, close this gap. Our framework brings the end-to-end Multi-Task Reinforcement Learning approach to the realm of legged robots, enabling the learned policy control any type of robot morphology. The key idea of our method is to allow the network to learn an abstract locomotion controller that can be seamlessly shared between embodiments thanks to our porphology-agnoste encoders and decoders. This flexible architecture can be seen as potential first step in building a founda- tion model for legged robot locomotion. Our experiments show that URMA can learn a locomotion policy on multiple embodiments that can be casily transferred unseen robot platforms in simulation and the real world.
Keywords: Locomotion, Reinforcement Learning, Multi-embodiment Learning
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Introduction
The robotics community has mastered the problem of robust gait generation in the last few years. With the help of Deep Reinforcement Learning (DRL) techniques, legged robots can show impres- sive locomotion skills. There are numerous examples of highly agile locomotion with quadrupedal robots [1, 2, 3, 4, 5, 6]. learning to run at high preedsjjumpinn over obstacles, walking on rough ter- rain, performing handstands, and completing parkour courses. Achieving these agile movements is often enabled by training in many parallelized simulation environments and using carefully tuned or automatic curricula on the task difficulty [7. 8]. Even learning simple locomotion behaviors directly on real robots possible but requires far more efficient learning approaches [9, 10]. Similar methods have been applied generate robust walking gaits for bipedal and humanoid robots [11, 12, 13]. The learned policies can be effectively transferred to the real world and work in all kinds of ter- rain with the help of extensive Domain Randomization (DR) [14, 15] during training. Additionally, techniques like student-teached teacher learning [1. 16] or the addition of model- components [17, 18] or constrains [19, 20, 21] to the learning process can further improve the learning efficiency and robustness of the policies.
At the same time, new advances in computational power, the availability of large datasets, and the development of foundation models are opening new frontiers for artificial intelligence, allowing us to implement and learn more complex and intelligent agent behaviors. Future robots will require incorporating these models into the control pipeline (22, 23]. However, to fully benefit from founda- tion models, we need to be able to integrate these high-level policies with the low-level control of the robots. The long-term objective would be to develop foundation models for locomotion, allowing zero-shot (or few-shot) deployment to any arbitrary platform. However, to reach this objective. is fundamental to adapt the underlying learning system to support different tasks and morphologies. Therefore, argue that the Multi-Task Reinforcement Learning (MTRL) problem is fundamental topic for the future research of robot locomotion, and, indeed, this formulation has recently attracted the interest of the community, using both structured [24] and end-to-end learning approaches [25]. MTRL algorithms share knowledge between tasks and learn a common representation space that can be used to solve all of them [26, 27J. To map differently sized observation and action spaces into and out of the shared representation space, implementations often resort to padding the observations and actions with zeros fit a maximum length [28] or using separate neural network head for each task [26]. These methods allow for efficient training but can be limiting when trying transfer to new tasks environments: for every new robot, the training process has to be repeated from scratch, as different embodiments require different hyperparameters, reward coefficients, training curricula, etc. Already in the case of the same robot morphology, e.g. quadrupeds, a trained policy can not be easily transferred when the number of joints is not the same for the robots. This is even more evident when trying to reuse the learned gait across different types of morphologies. This issue closely related the fundamental correspondence problem in robotics [29], the policy has to learn an internal mapping between the different action and observation spaces and the embodiments themselves, which define the robots' kinematics. In practice, the number of joints and feet of legged robot determines the size of its action and observation space, which can differ for every new robot. This often prevents a straightforward transfer of existing policies as the learning architecture fully depends on the specific robot platform.
To tackle this problem and to move to more powerful and general policies that can be used as locomotion foundation models, we propose a novel MTRL framework that allows simultaneously learning locomotion tasks with many different morphologies easily and effectively. Our approach is based on a novel neural network architecture that can handle differently sized action and observation spaces, allowing the policy to adapt easily to diverse robot morphologies. Furthermore, our method allows us zero-shot deployment of the policy to unseen robots and few-shot fine-tuning on novel target platforms. We highlight the effectiveness of our approach, first with a theoretical analysis and then by training a single locomotion policy on 16 robots, including quadrupeds, hexapods, bipeds, and humanoids. Finally, shot transfer the learned policy to two simulated and three real- world robots, showing the transferability and robustness of our method.

---

Related Work
Early work on controlling different robot morphologies is based on the idea of using Graph Neural Networks (GNNs) to capture the morphological structure of the robots [30, 31, Each node the graph represents a joint of a robot, and its state is comprised the joint's specific information, e.g. current position, velocity, etc. Through message passing, the GNN can then aggregate information from neighboring joints and learn to control the robot control different robots even when removing some of their limbs, but they struggle to generalize to many different morphologies at once, as every morphology requires different graph structure. Furthermore, the local nature of message passing can lead to information bottlenecks in the policies and the inability to act as a cohesive global controller [33].
whole. GNN-based approaches can
Transformer- baasd architectures have been proposed to overcome the limitations of GNNs by using the attention mechanism to globally aggregate information of varying numbers of joints [33, 25, 34, 35J. These methods still lack substantial generality as they are limited to morphologies that were defined priori. For example, Kurin et al. [33] uses incoder-decoded pairs for each type of joint, which limits the architecture to a set of predefined joint types and does not allow for components that only have associated observations but no actions, e.g. the feet of legged robots. Trabucco et al. [25] defines tokens for each type of observation and joint, which is a more general system, but those tokens are handcrafted, and there is different set joint tokens for every morphology. which makes generalization between morphologies and to new ones difficult even impossible.
Besides the GNN- and Transformer-bast methods, which consider only environments with 2D- planar or physically implausible robots, Shafiee et al. [24] recently showed that a single controller can be trained to control 16 different 3D-simulated quadrupedal robots and to transfer to two of them in the real world. Their method uses a Central Pattern Generator (CPG) and inverse kinematics to generate and track trajectories of the four feet of the robots. This approach has the drawback of discarding the joint-specific information, and the controller can be deployed only on robots with the same number of feet. Furthermore, Feng et al. [36] and Luo et al. [37] use procedurally generated quadrupeds in simulation to train single policy that transfers to unseen real-world quadrupeds. However, their learning framework assumes that every robot is in the same morphology class, and their neural network architecture can only deal with robots with the same number of joints. Com- pared to all the other approaches, our method can handle multiple embodiments from any legged morphology, can adapt to arbitrary joint configurations with the same network and can be deployed on real world robots.
Multi-embodiment Locomotion with a Single Policy
In MTRL the objective is to learn a single policy To that optimizes the average of the expected discounted return J (0) over the reward function m across M tasks:
M
J(0)- M Σ ('),),
TThh (0) = R
Σ 2 77 (s, (0
(1)
where 7 a trajectory given by the state-action pairs (se, (t), 7 is the discount factor, and T is the time horizon. In our case, we consider different robot embodiments as separate tasks and train a policy controlling all robots and optimizing the objective described in (I), We aim to design a policy where the underlying neural network architecture is independent of the set of possible embodiments. Therefore, we propose the Unified Robot Morphology Architecture (URMA), which is completely morphology agnostic, i.c. it can be applied to any type of robot with any number of joints such that there is no need to define the possible morphologies or joints beforehand. We use URMA to learn robust locomotion policies, but its formulation is general enough to be applied to any control task, Figure presents schematic overview of URMA. In general, URMA splits the observations of a robot into distinct parts, encodes them with a simple attention encoder [38] with a learnable temperature [39], and uses universal morphology decoder to obtain the actions for every joint of the robot.
3

---

JONE encoder
Core network
Univertal decodar
Laye timm
Uneer
Taye Nurk
Siack
Luce
Bill
(Reerio socediing
Figure 2: Overview of URMA. Left Joint observations and descriptions are encoded and com- bined into a single joint latent vector through an attention head. Bottom center - Feet observations and descriptions observations are fed through the core network to get the action latent vector. Right - The universal morphology decoder encodes the joint descriptions and pairs them with the action latent vector and the single joint latent vector to produce the action mean and standard deviation for the final action.
encoded in the same way. Top center Joint latent, feet latent, and general
To handle observations of any morphology, URMA first splits the observation vector into robot- specific and general observations Og. where the former can be of varying size, and the latter has a fixed dimensionality. For locomotion, we subdivide the robot-specific observations into joint and feet-specific observations. This split is not necessary but makes the application to locomotion cleaner. In the following text, we describe everything w.r.t. the joint-specific observations, but the same applies to the feet-specific ones well. Every joint of robot is composed of joint-specific ob- servations oj and description vector dj. which is fixed-size vector that can uniquely describe the joint by using characteristic properties like the joint's rotation axis, its relative position in the robot, torque and velocity limits, control range, etc. (see Appendix A for more details). The description vectors and joint-specific observations are encoded separately by the Multilayer Perceptrons (MLPs) So and f and are then passed through a simple attention head, with a learnable temperature r and a minimum temperature e, to get a single latent vector
fold))
exp
Ejoints
Σ 25 JEJ
2j
T T+E
fald, T+e
fo(o,);
Ejer exp
that contains the information of the joint-specific observations of all joints. With the help of the attention mechanism, the network can learn to separate the relevant joint information and precisely route it into the specific dimensions of the latent vector by reducing the temperature of the softmax close to zero. The joint latent vector žjoints is then concatenated with the feet latent vector ž and the general observations Og and passed to the policies core MLP hg to get the action latent vector Zaction = ho Sjoims, 2. To obtain the final action for the robot, we use our universal morphology decoder, which takes the general action latent vector and pairs it with the set of encoded specific joint descriptions and the single joint latent vectors to produce the mean and standard deviation of the actions for every joint, from which the final action is sampled
al ~ (uv Eaction 2,), (d)()))
(d,))
(3)
To ensure that only fully normalized and well- observations come into the network, we use LayerNorm [40] after every input layer. The learning process also benefits from adding another LayerNorm in the action mean network μy. We argue that this choice improves the alignment of the different latent vectors entering into μ4, better. To ensure a fair comparison, we also use LayerNorms with the same rationale in the baseline architectures. We highlight the benefits in the ablation study on the usage of LayerNorm for URMA in Figure 4 of Appendix D.
Our second contribution the open-source modular learning framework, which enables to easily train robust and transferable locomotion policies for all kinds of legged robots. When adding robot the training set, only the reward coefficients, controller gains, and domain randomization

---

ranges have to be adjusted, which can be easily done by slightly modifying the ones from existing robots in the framework. As the penalty terms in the reward function are not essential for learning the core locomotion but only shape the resulting gait, we apply a time-dependent fixed-length cur- riculum r1) = min (f, 1) r where is the current training step, the curriculum length, and rz is the final penalty coefficient. This speeds up the learning and makes the coefficient tuning pro- cess easier and more forgiving, as the policy can handle higher penalties better when it has already learned to perform basic locomotion. The full details the environment setup, reward function, and domain randomization can be found in Appendix A.
Theoretical Analysis
To evaluate the benefit of learning shared representations across robot morphologies through URMA, we extend the multi-task risk bounds from Maurer al. [41] and D'Eramo al. [26] to our morphology-agnostic encoder and decoder. As will use the Proximal Policy Optimization (PPO) algorithm in our learning framework, we frame our problem as an evaluation of the perfor- mance difference between the PPO training the empirical dataset and the optimal policy update with infinite samples. In our simplified analysis, we will assume that our policy optimization step can find the policy that minimizes the surrogate loss over the current dataset; ii the trust region is small enough such that the surrogate loss and the expected discounted return of the policy are close enough; iii. the surrogate loss is computed with the advantage. Given our of robot embod- iments μ (J1,-.. (M,) the set X M of input samples from the space of observations, descriptions and actions X' for each of the M tasks and the corresponding set Y E RR of advan- tages A divided by the initial probabilities we can describe the policy learning as finding the minimizer f. h, vó of the following optimization problem:
(f.h,
min (f...))
el Fe
(4)
with f F X RR x
H: RKK x X and E W R x X' R being
the encoder, core and decoder networks, and é: R R [0, 1] the normalized policy optimization loss function. We quantify the performance of the functions h, w with the task-averaged risk
M
e (f.h. w) = M Σ
Exxyripp (((((((()))) Y).
(5)
We define Eive as the minimum of this risk, with the minimizers f", h² and w". We mea- sure the complexity of some function class Z composed of K functions via the set 2(X) = {(Ex (Xmi)) : E 2} S IR KM with the Gaussian complexity [42]
G(Z(X)) = sup
Ti (Xmi)
(6)
where 7mki are i.i.d. standard Gaussian random variables.
Furthermore, we define T(2), as the upper bound of the Lipschitz constant of all = in 2, and the Gaussian average of Lipschitz quotients
(7, zw) =(>'))
O(2)=
sup supsez
v.
11y v'll
where y vector of d i.i.d. standard Gaussian random variables and 2 E 2 Y + Rd with
Y C RP. Using the definitions above, we bound the risk Eavg:
Theorem 1. Let p. F, H and W be defined as above and assume 0 H. and w(0) = 0, Ve E W.
Then for ā > 0 with probability at least 1 8 in the draw of (X, Y)
I M
II Ph we have that
Eang (f. h @) Ser (()IL))()))(((X nM
L(E)L(W) supp (I (()) nM
L(E) P G(H(p)) +
T.(E) sup,, (K((())) nM
8 (n((2)
nM
nM
(8)
5

---

The proof of Theorem 1 and discussion on the assumptions can be found in Appendix M. For reasonable function classes W, the Gaussian average of Lipschitz quotients O(W) can be bounded independently from the number of samples. For most settings, the Gaussian complexity G(F(X)) OV).M). Also the terms sup, supyll](() [X)] and supa,, s [|][][())]]]] are O(vnM), ) if they can be uniformly bounded. Using these assumptions, the URMA policy structure better suited for multi- task learning all the first four terms are O (3/v5NT) In comparison, the multi-head architecture from from D' Eramo et al. [26] requires additional encoder and decoder heads for every task, and thus, the cost of learning all the encoders and decoders is only O (1/F), i.e. it is not reduced when increasing the number of tasks M, as it is in our case for URMA. In conclusion, as URMA only uses a single general encoder and decoder for all tasks, it compares favorably to the typical multi- head approach as can focus on learning only a single mapping to the shared representation space compared to the multi-head architecture which needs learn M different encodings and decodings. This leads the lower sample cost of learning these shared representations.
Experiments
In this section, evaluate method from three different perspectives. First, we assess how well the model learns to control multiple embodiments in parallel against classical MTRL baselines. Then, we analyze how well the URMA architecture performs in terms of zero and few-shot transfer. Finally, we test the deployment capabilities of our learning framework and control architecture on real robots, allowing us to bridge the sim-to-real gap effectively. For all experiments in simulation, we will use the following two baselines.
Multi-Head Baseline. One way of dealing with the issue of different action and observation space sizes is to use multi-head architecture with an encoder head for every environment, shared core, and a decoder head for every environment [26]. We implement this baseline by using one shared encoder and decoder for all quadruped robots, one for all humanoid and bipedal robots, and one for the hexapod. The observations of all robots with identical morphology are arranged in the same order for the encoder, and the observations for missing joints are simply set to zero, e.g. humanoids often have different joint configurations. Compared to URMA, this baseline allows for efficient learning as the morphology-specine heads inherently separate the observations and arrange them in the correct order from the beginning. However, introducing new joints or completely new morphologies requires adding new neurons to a head or training a completely new head from scratch. Every additional head is a new mapping into and out of the shared representation space, which leads to higher learning complexity compared to URMA.
Padding Baseline. Another way to handle differently sized action and observation spaces is to pad the observations and actions with zeros to fit maximum length [28]. Therefore, a specific observation dimension can now represent different things for different robots. We add one-hot task ID to the observations to ensure that the policy can distinguish between the robots, as typically done in MTRL [27]. Compared to URMA, the padding baseline is less complex but has similar issues when transferring to new morphologies like the multi-head baseline, as a new robot essentially represents a completely new task, and the policy has hard time transferring knowledge between the differently structured observations and actions.
To train our locomotion policy, we use the CPU-based MuJoCo physics simulator [43] for 16 dif- ferent simulated robots with three learning environments each, resulting in 48 parallel environments in total. Figure 1 shows all the simulated robots we utilize for training and the three real robots that we use for deployment in the real world. The set of robots includes nine quadrupeds with three different joint configurations, five humanoids with five different joint configurations, one biped, and one hexapod. To leverage the huge amounts of data that we can generate in simulation, we use the PPO algorithm [44]. We build on the codebase of the DRL library RL-X [45] to implement our architecture and the baselines in JAX and to run the experiments. All hyperparameters used for URMA and the basclines are noted in Table 4 in Appendix B.

---

Results
First, we want to evaluate the training efficiency of MTRL our setting. We train URMA and the baselines on all robots simultaneously and compare the average return the single-robot training setting, where separate policy is trained for every robot. All policies are trained 100 million steps per robot, and every experiment shows the average return over 5 seeds and the corresponding 95% confidence interval. Additionally, we plot the empirical maximum performance when contin- uing the training for 1.6B steps only single robot as dashed line. The performance on all 16 robots individually can be found in Figure 13 in Appendix L.. Figure 3 confirms the advantage in learning efficiency MTRL over single-task learning, as URMA and the multi-head baseline learn significantly faster than the average over training only on single robot at time. As expected, early on in training, URMA learns slightly slower than the multi-head bascline due to the time needed by the attention layers to learn to separate the robot-specifee information, which the multi-heat baseline inherently does from the beginning. However, URMA ultimately reaches higher final performance. The padding baseline performs noticeably worse than the other two. We argue that the policy has trouble learning the strong separation in representation space between the different robots robots-which necessary for the differently structured observation and action spaces oonly
only based on the task ID.
Next, evaluate the zero-shot and few-shot transfer capabilities of URMA and the baselines two robots that were withheld from the training set of the respective policies. We test the zero-shot transfer on the Unitree A1, robot whose embodiment is similar to other quadrupeds in the training set. Figure 3 shows the evaluation for the Al during training process with the other 15 robots and highlights that both URMA and the multi-head baseline can transfer perfectly well to the A1 while never having seen during training. The policy trained only on the Al (shown in black) performs distinctly worse during the 100M training steps as it needs more samples per robot to learn the task. It eventually catches up the multi-embodiment zero-shot performance while training for 1.6B steps directly on the A1. Further ablations the minimal set of quadrupeds needed for good zero-shot capabilities on the A1 can be found in Appendix H.
Training on all 16 robots
Zero shot on Unitree A1
60
50
Average return
return
40
40
30
20
20
0.8
1.0
0.4
0.6
0.8
1.0
Environment steps per robot
le8
Environment steps per robot
108
Zero-shot and fine-tuning MAB Silver Badger
Zero-shot with foot observation dropout
40
Average return
40
Jurerage return
30
30
20
20
20
1.2 168
1.0
Environment steps per robot
Environment steps per robot
1s8
URMA
Multi-Head Baseline
Padding Baseline
Single-robot training
Single-robat training at 1.6B steps
Figure 3: Top left- Average return of the three architectures during training on all 16 robots com- pared to the single-robot training setting. Top right Zero-shot transfer to the Unitree A1 while training on the other 15 robots. Bottom left- Zero-shot transfer to the MAB Robotics Silver Badger while training on the other 15 robots and fine-tuning on only the Silver Badger afterward. Bottom right- - Zero-shot evaluation on all 16 robots while removing the feet observations.
7

---

To investigate an outto--distribution embodiment, we use the same setup as for the Al and evaluate zero-shot on the MAB Robotics Silver Badger robot, which has an additional spine joint in the trunk and lacks feet observations, and then fine-tune the policies for 20 million steps only on the Silver Badger itself. Figure 3 shows that URMA can handle the additional joint and the missing feet observations better than the baselines and the only method capable of achieving a good gait at the end of training. After starting the fine-tuning (gray vertical line), URMA maintains the lead in the average return due to the better initial zero-shot performance. Similar zero-shot and fine-tuning experiments for the humanoid morphology can be found in Appendix I.
To further assess the adaptability of our approach, we evaluate the zero-shot performance in the setting where observations are dropped out, which can easily happen in real-world scenarios due sensor failures. To test the additional robustness in this setting, train the architectures on all robots with all observations and evaluate them all robots while completely dropping the feet observations. Figure 3 confirms the results from the previous experiment with the Silver Badger and shows that URMA can handle missing observations better than the baselines.
Finally, we deploy the same URMA policy on the real Unitree AI, MAB Honey Badger, and MAB Silver Badger quadruped robots. Figure and the videos on the project page show the robots walking with the learned policy on pavement, grass, and plastic turf terrain with slight inclinations. Due to the extensive DR during training, the single URMA policy trained 16 robots in simulation can be zero-shot transferred to the three real robots without any further fine-tuning. While the Unitree A1 and the MAB Silver Badger are in the training set, the network is not trained on the MAB Honey Badger. Despite the Honey Badger's 's gait not being as good as the other two robots, it can still locomote robustly on the terrain we tested, proving the generalization capabilities of our architecture and training scheme.
Limitations. While method is the first end-to-end approach for learning
locomotion, many open challenges remain. On one side, our generalization capabilities rely mostly on the availability of data, therefore zero-shot transfer to embodiments that are completely out of the training distribution is still problematic. This issue could be tackled by exploiting other techniques in the literature, such as data augmentation and unsupervised representation learning, to improve our method's 's's generalization capabilities. Furthermore, we currently omit exteroceptive sensors from the observations, which can be crucial to learning policies that can navigate in complex environments and fully exploit the agile locomotion capabilities of legged robots. Lastly, as humanoid robots only recently started to be available for reasonable prices, we could not test the deployment these platforms yet. Better modelling, more reward engineering or additional randomization might be necessary to ensure their real-world transfer.
of
4 Conclusion
We presented URMA, a open-source framework' 1 to learn robust locomotion for different types of robot morphologies end-to-end with a single neural network architecture. Our flexible learning framework and the efficient encoders and decoders allow URMA to learn a single control policy for 16 different embodiments from three different legged robot morphologies. We highlight URMAs learning efficiency in theoretical analysis of its task-averaged risk bound and compare it to prior work. In practice, URMA reaches higher final performance on the training with all robots, shows higher robustness to observation dropout, and better zero-shot capabilities to new robots compared to MTRL baselines. Furthermore, we deploy the same policy zero-shot on two known and one unseen quadruped robot in the real world. We argue that this multi-embodimmet learning setting can be easily extended to more complex scenarios and can serve as a basis for locomotion foundation models that can act on the lowest level of robot control. Finally, the URMA architecture is general enough be applied to not only any robot embodiment but also any control task, making task generalization, also for non-locomotion tasks, interesting avenue for future research.
Prooecct page: https::/nico-bohllaae
run them_ all_wellllllllll wel_wtesite/
8

---

Acknowledgments
This project was funded by National Science Centre, Poland under the OPUS call in the Weave programme RMM-2021/43///T6000000 11, and by the German Science Foundation (DFG) under grant number PE 2315/17-1. Part of the calculations were conducted on the Lichtenberg high performance computer at TU Darmstadt.
References
[1] T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning robust per- ceptive locomotion for quadrupedal robots in the wild. Science Robotics, 7(62))eab 2022.
[2] G. B. Margolis and P. Agrawal. Walk these ways: Tuning robot control for generalization with multiplicity of behavior. In Conference on Robot Learning, pages 22-31. PMLR, 2023.
[3] S. Choi, G. Ji, J. Park, H. Kim, J. Mun, J. H. Lee, and J. Hwangbo. Learning quadrupedal locomotion on deformable terrain. Science Robolics, 8(74))eade2256, 2023.
[4] K. Caluwaerts, A. Iscen, J. C. Kew, W. Yu, T. Zhang, D. Freeman, K.-H. Lee, L. Lee, S. Sal- iceti, V. Zhuang, al. Barkour: Benchmarking unimal-level agility with quadruped robots. arXiv preprint arXiv:2305. 14654, 2023.
[5] Z. Zhuang, Z. Fu, J. Wang. C. Atkeson, S. Schwertfeger, C. Finn, and H. Zhao. Robot parkour learning. In Conference on Robot Learning (CoRL), 2023.
[6] X. Cheng, K. Shi, A. Agarwal, and D. Pathak. Extreme parkour with legged robots. In Robo- Letics: Workshop on Robot Learning in Athletics@ CoRL 2023. 2023.
[7] N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning walk in minutes using massively parallel deep reinforcement learning. In Conference Robot Learning, pages 91-100. PMIR, 2022.
[8] G. Margolis, G. Yang, K. Paigwar, T. Chen, and P. Agrawal. Rapid locomotion via reinforce- ment learning. Robotics: Science and Systems, 2022.
[9] L. Smith, I. Kostrikov, and S. Levine. A walk in the park: Learning walk in 20 minutes with model-free reinforcement learning, arXiv preprint arXiv:2208.0086// 2022.
[10] L. Smith, Y. Cao, and Levine. Grow your limits: Continuous improvement with real-world for robotic locomotion, arXiv preprint arXiv:2310.1765 2023.
[11] J. Siekmann, K. Green, J. Warila, A. Fern, and J. Hurst. Blind bipedal stair traversal via sim- to-real reinforcement learning. In Robotics: Science and Systems, 2021.
[12] A. Kumar, Z. Li, J. Zeng. D. Pathak, K. Sreenath, and Malik. Adapting rapid motor adapta- tion for bipedal robots. In 2022 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 1161-1168. IEEE, 2022.
[13] I. Radosavovic, T. Xiao, B. Zhang, Darrell, J. Malik, and K. Sreenath. Real-world humanoid locomotion with reinforcement learning. arXiv:2303.0333 2023.
[14] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA), pages 3803-3810, IEBE, 2018.
[15] L. Campanaro, S. Gangapurwala, W. Merkt, and Havoutis. Learning and deploying robust locomotion policies with minimal dynamics randomization. arXiv preprint arXiv:2209.12888 12878, 2022.

---

[16] A. Kumar, Z. Fu, D. Pathak, and J. Malik. Rma: Rapid motor adaptation for legged robots. Robotics: Science and Systems XVII, 2021.
[17] F. Jenelten, J. He. F. Farshidian, and M. Hutter. Dtc: Deep tracking control-a unifying ap- proach to model-based planning and reinforcement-lerriing for versatile and robust locomo- tion. arXiv preprint arXiv:2309. 15462, 2023.
[18] M. Kasaei, M. Abreu, N. Lau, A. Pereira, and L. P. Reis. A epg-based agile and versatile locomotion framework using proximal symmetry loss, arXiv preprint arXiv:2103.0000 2021.
[19] Lee, L. Schroth, V. Klemm, M. Bjelonic, Reske, and M. Hutter. Evaluation of constrained reinforcement learning algorithms for legged locomotion. arXiv preprint arXiv:2309. 2023.
[20] Y. Kim, Oh, J. Lee, Choi, Ji, M. Jung, D. Youm, and Hwangbo. Not only rewards but also constraints: 2024.
Applications on legged robot locomotion. IEEE Transactions on Robotics,
[21] He, C. Zhang, W. Xiao, G. He, C. Liu, and G. Shi. Agile but safe: Learning collision- free high-speed legged locomotion. arXiv preprint arXiv:2401. 17583, 2024.
[22] Y. Ouyang, J. Li, Y. Li, Z. Li, C. Yu, K. Sreenath, and Y. Wu. Long-horizon locomo- tion and manipulation on a quadrupedal robot with large language models. arXiv preprini arXiv:2404.05291, 2024.
[23] W. Song. H. Zhao, P. Ding, C. Cui, S. Lyu, Y. Fan, and D. Wang. Germ: A generalist robotic model with mixture-of-expert. for quadruped robot. arXiv preprint 1rX3:22403.13358, 2024.
[24] M. Shafiee, G. Bellegarda, and A. ljspeert. Manyquadrupeds: Learning single locomotion policy for diverse quadruped robots. arXiv preprint arXiv:2310. 2023.
[25] B. Trabucco, M. Phiclipp, and G. Berseth. Anymorph: Learning transferable polices by in- ferring agent morphology. In International Conference on Machine Learning, pages 21677- 21691. PMLR, 2022.
[26] C. D'Eramo, D. Tateo, A. Bonarini, M. Restelli, Peters, et al. Sharing knowledge in multi-task deep reinforcement learning. In 8th International Conference on Learning Representations, {ICLR) 2020, Addis Ababa, Ethiopia, April 26-30, 2020, pages 1-11. In- ternational Conference on Learning Representations, ICLR, 2020.
[27] A. Hendawy, Peters, and D'Eramo. Multi-task reinforcement learning with mixture of orthogonal experts. In The Twelfth International Conference on Learning Representations, 2023.
[28] Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pages 1094-1100. PMLR, 2020.
[29] C. L. Nehaniv and K. Dautenhahn. The correspondence problem. In Imitation in animals and artifacts, pages 41 11. MIT Press, 2002.
[30] T. Wang, R. Liao, J. Ba, and S. Fidler. Nervenet: Learning structured policy with graph neural networks. International conference on learning representations, 2018.
[31] W. Huang, I. Mordatch, and D. Pathak. One policy to control them all: Shared modular policies for agent-agnostic control. In International Conference on Machine Learning, pages 4464- PMLR, 2020.
[32] J. Whitman, M. Travers, and H. Choset. Learning modular robot control policies. 1EEE Transactions Robotics, 2023.
10

---

[33] V. Kurin, M. Igl, T. Rocktäschel, J. Böhmer, and S. Whiteson. My body is cage: the role of morphology in graph-based incompatible control. In International Conference on Learning Representations (ICLR), 2021.
[34] A. Gupta, L. Fan, S. Ganguli, and L. Fei-Fei. Metamorph: learning universal controllers with transformers, In International Conference on Learning Representations. ICLR, 2022.
[35] B. Li, H. Li, Y. Zhu, and D. Zhao. Mat: Morphological adaptive transformer for universal morphology policy learning. IEEE Transactions Cognitive and Developmental Systems, 2024.
[36] G. Feng, H. Zhang, Z. Li, X. B. Peng, B. Basireddy, L. Yue, Song, L. Yang, Liu, K. Sreenath, et al. Genloco: Generalized locomotion controllers for quadrupedal robots. In Conference Robot Learning, pages 1893-1903. PMLR, 2023.
[37] Z. Luo, Y. Dong, X. Li, R. Huang, 2. Shu, E. Xiao, and P. Lu. Moral: Learning morphologically adaptive locomotion controller for quadrupedal robots on challenging terrains. IEEE Robotics and Automation Letters, 2024.
[38] D. Bahdanau, K. H. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, 2015.
[39] J. Lin, X. Sun, X. Ren, M. Li, and Q. Su. Learning when to concentrate divert attention: Self Self-adaptive attention temperature for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2985-2990. Asso- ciation for Computational Linguistics, Oct.-Nov. 2018.
[40] J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450. 2016.
[41] A. Maurer, M. Pontil, and B. Romera-Paredes. The benefit of multitask represennatiionn learning. Journal of Machine Learning Research, 17(81):1-32, 2016.
[42] L. Bartlett and Mendelson. Rademacher and gaussian complexities Risk bounds and structural results. Journal of Machine Leurning Research, 3(Nov):463-482 2002.
[43] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model- control. In 2012 JEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012.
[44] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 170707637777777 2017.
[45] N. Bohlinger and Dorer. Rl-x: deep reinforcement learning library (not only) for robocup. In Robot World Cup, pages 228-239. Springer, 2023.
[46] G. Ji, J. Mun, H. Kim, and J. Hwangbo. Concurrent training of control policy and a state estimator for dynamic and robust legged locomotion. IEEE Robotics and Automation Letters, 7(2):460044637, 2022.

---

Appendix
A Characteristics of the Learning Environment and Robot Embodiments
Environment Details. Every robot is controlled with PD controller at 50Hz, where the target joint velocity is set to 0, and the target joint position is calculated by: g gaominal + aa with quominal being a nominal joint position, the action of the policy and aa scaling factor. We inherit this action scheme from the implementation details from the DRL locomotion literature [2, 7, 16]. The gains kp and ky and the scaling factor %a vary from robot to robot and are noted Table 1. The observation space is divided into joint-specific, feet-specific, and general observations. The joint-specific observations Oj consist of the joint position q, joint velocity q, and previous action a for each joint. The feet observations of include the contact flag Py and the time since each foot's last contact pJ. The general observations 0g the linear velocity of the trunk/torso (only for the critic), angular velocity w, command x, and yaw velocities (re-sampled twice per episode on average), orientation of the gravity vector current height above ground h (only for the critic), PD controller variables kp. ky and da. total mass of the robot m and its dimensions: length L. width W and height H. All observations are concatenated to o = [o,, of. 0g) and normalized [-1, 1]. For URMA, the joint descriptions d, consist of a joint-specific part, which is made of the relative 3D position in the nominal joint configuration, rotation axis, number of direct child joints, joint nominal position, torque limit, velocity limit, damping, rotor inertia, stiffness, friction coefficient and min and max control range, and a general robot part, which is made of kpp ka, aa m. L, W, H. Similarly, the feet descriptions dy consist of the relative 3D position of the foot in the nominal joint configuration and of the general robot attributes kp. ka, Oa m, L,, W, H.
Morphology
Robot name
kp
ky
0
m
L
W
Quadruped
ANYbotics ANYmal
80.0
2.0
0.5
33.3
1.21
0.81
1.37
ANYbotics ANYmal C
12
80.0
2.0
0.5
45.0
1.09
0.88
0.91
Google Barkour vO
12
20.0
0.5
0.25
11.5
0.70
0.38
0.45
Google Barkour vB
12
30.0
0.5
0.25
11.5
0.73
0.48
0.49
MAB Silver Badger
13
20.0
0.5
0.25
13.12
0.78
0.41
0.52
Petoi Bittle
25.0
0.5
0.6
0.2
0.17
0.12
0.12
Unitree A1
12
20.0
0.5
0.25
12.5
0.67
0.43
0.48
Unitree Gol
12
20.0
0.5
0.25
12.7
0.70
0.42
0.48
Unitree Go2
12
20.0
0.5
0.3
15.2
0.75
0.44
0.48
Biped
Agility Robotics Cassie
10
70.0
2.0
0.6
33.3
0.60
0.60
1.26
Humanoid
PAL Robotics Talos
24
80.0
2.0
0.75
93.3
0.46
1.10
1.65
Robotis OP3
20
21.0
0.5
0.6
3.1
0.24
0.28
0.53
SoftBank Nao V5
22
30.0
0.5
0.6
5.3
0.17
0.43
0.59
Unitree GI
23
45.0
1.0
0.5
32.2
0.29
0.55
1.26
Unitree HI
19
60.0
2.0
0.75
51.4
0.55
0.83
1.77
Hexapod
Custom Hexapod
18
30.0
0.5
0.6
1.9
0.43
0.56
0.24
Table I: Morphology type, robot name, number of controlled joints (J), gains, action scaling factor, total mass in kg. length in meter, width in meter and height in meter for every robot in the training set.
Reward Function. The reward function consists of two tracking reward terms for following the command velocities e and multiple penalty terms to shape the gait. All the terms are scaled with individual coefficients, summed up, and clipped to be above zero. Table 2 and Table 3 note the equations for each term and the respective coefficients.
12

---

Term
Equation
(11) Xy velocity tracking
exp(
Uzy
Cayl (anl2/0.22 2/0.25)
(T2) Yaw velocity tracking
exp(
Craw Soawl2/0... P/0.25)
T3)) velocity penalty
|va
(T4) Pitch-roll velocity penalty
leppect. roll
(T5) Pitch-roll position penalty
Pinch roll 2
(T6) Joint nominal differences penalty
mocninal (2
(T7) Joint limits penalty
10.94mmi
< 0.94mmxx)
(T8) Joint accelerations penalty
(T9) Joint torques penalty
|r|
(T10) Action rate penalty
d
(TI1) Walking height penalty
h Recomnaal
(T12) Collisions penalty
Peollisions
(T13) Air time penalty
Ep)p)) 0.5)
(T14) Symmetry penalty
E, "p1"""")"(p) (ibbe))
Table 2: Reward terms that make up the reward function. The coefficients for each term differ for every robot and are noted in Table 3.
Robot
T1
T2
T3
T4
T5
T6
T7
T8
T9
T10
T11
T12
T13
T14
T
ANYmalB
2.0
1.0
2.0
0.05
0.2
0.0
10.0
2.5e-7
2--4
0.01
30.0
1.0
0.1
0.5
20e6
ANYmalC
2.0 1.0
2.0
0.05
0.2
0.0
10.0
2.5e-7
2e-4
0.01
30.0
1.0
0.1
0.5
20e6
Barkour vO
3.0 1.5 2.0 0.05
0.2
0.0
10.0 2.5e-7
2e-4
0.01 30.0
1.0
0.1
0.5
15e6
Barkour vB
2.0 1.0 2.0 0.05 0.2
0.0 10.0 2.5c-7 2e-4
0.01 30.0
1.0
0,1
0.5
15e6
Silver Badger
2.0 1.0 2.0 0.05 0.2 0.0 10.0 2.5e-7
2e-4 0.01 30.0
1.0
0.1
0.5
12c6
Bittle
5.0 2.5 2.0 0.05 0.2 0.0 10.0 2.5e-7 2e-4 0.01 30.0
1.0
0.1
0.5
40e6
AI
2.0 1.0 2.0 0.05 0.2 0.0 10.0 2.5e-7 2c-4 0.01 30.0
1.0
0.1
0.5
12e6
Gol
2.0 1.0 2.0 0.05 0.2 0.0 10.0 2.5c-7 2c-4 0.01 30.0
1.0
0.1
0.5
12c6
Go2
2.0 1.0 2.0 0.05 0.2 0.0 10.0 2.5e-7 2e-4 0.01 30.0
1.0
0.1
0.5
12e6
Cassie
3.0 1.5 2.0 0.05 0.2 0.0 10.0 2.5e-7 2e-5 0.01 30.0
1.0
0.1
0.5
50e6
Talos
4.0 2.0
2.0
0.05 0.2
0.2 10.0 2.5e-7 2e-5 0.01 30.0
1.0
0.1
0.5
80e6
OP3
4.0 2.0
2.0 0.1 0.2 0.4 10.0 1.2e-6 4e-4 6e-3 30.0
0.1
0.5
40e6
Nao V5
4.0 2.0
2.0
0.1 0.2 0.15 10.0 1.2e-6 4e-4 6e-3 30.0
1.0
0.1
0.5
40e6
GI
3.0 1.5 2.0 0.05
0.2
0.2 10.0 2.5e-7 5e-5 0.01 30.0
1.0
0.1
0.5
50e6
H1
2.0
1.0 2.0
0.05
0.2 0.2 10.0 2.5e-7 2e-5 0.01 30.0
1.0
0.1
0.5
50e6
Hexapod
4.0
2.0 2.0 0.05 0.2 0.0 10.0 2.5e-7 2e-4 0.01 30.0
1.0
0.1
0.5
15e6
Table 3: Reward coefficients Ye for all the reward terms for all robots. The last column T is the dependennndent reward curriculum length until all penalties are linearly increased from zero their final values.
13

---

Domain Randomization. As known from previous work, the extensive application of Domain Randomization is necessary to ensure that learned policies can be transferred to the real world [1, 13,40 46]. Therefore, we randomize as many aspects as possible to learn a robust policy for all robots in training set and also to ensure that the policy can generalize well to otherwise unseen parameters during zero-shot transfer new robots. We randomize the robot attributes (trunk mass and inertia, center of mass displacement, feet sizes, torque anmits,ooim velocity limits, joint dampings, rotor inertias, joint stiffnesses, joint friction coefficients and joint control ranges.), the terrain (ground friction coefficient, gravity, contact stiffness and damping), the control (PD gains, action scaling factor, motor strength, joint offsets and actuator delays) and the initial state of the robot (linear and angular velocity, orientation, joint positions and velocities). A new set of these variables is sampled twice per episode on average. Also, to make the policy robust to noisy sensors and learn to act even when some observations are missing, we add observation noise every time step and randomly drop observations, i.e. set them to zero. Finally, make the policy robust external disturbances, randomly perturb the linear velocity of the robots.
B Hyperparameters
The hyperparameters used for training the policies in our experiments are noted in Table 4. We use the same hyperparameters for all architectures to ensure a fair comparison. For the fine-tuning experiments, we reduce the learning rate by 1/3 to make the learning process more stable and avoid forgetting in the policy by initial overfitting the new robot.
Hyperparameter
Value
Batch size
522240 (16 robots * 3 envs 10880 steps)
Mini-batch size
32640 (16 robots 2040 samples)
Nr. epochs
10
Initial and final learning rate
0.0004, 0.0 (Annealed over 100M steps)
Entropy coefficient
0.0
Discount factor
0.99
GAE A
0.9
Clip range
0.1
Max gradient norm
5.0
Initial action standard deviation
1.0
Clip range action standard deviation
1e-8, 2.0
Clip range action mean
-10.0, 10.0 (Before applying o.)
Nr. policy network parameters
430k (+/- 10k for all architectures)
(URMA-specific) Initial softmax temperature
1.0
(URMA-specific) Minimum softmax temperature
0.015
Table 4: The PPO hyperparameters that are used all experiments for training.
C Training Time and Hardware
All multi-embodiment learning experiments conducted AMD Ryzen AMD EPYC CPUs and NVIDIA RTX 3090 or NVIDIA A5000 GPUs. For single training run with all 16 robots and the hyperparameters noted in Table 4, we allocate 20 CPU cores, 32GB of RAM and GPU. With this setup, training the policy for 100 million steps per robot (1.6 billion steps in total) takes around 2.5 days. The CPU-based MuJoCo simulation with the small amount of parallel environments (48) poses the bottleneck our training process. Moving to a more powerful GPU-based simulator with thousands parallel environments would significantly speed up the training.
14

---

D Ablation on LayerNorm
To evaluate the effectiveness of the LayerNorm layers in the URMA architecture, we perform an ablation study where we remove LayerNorm layers from the architecture.
Training on all 16 robots
50
Awerage return
40
30
20
0.0
0.2
0.4
0.6
0.8
1.0
Environment steps per robot
lell
URMA
URMA without LayerNorm
Figure 4: Ablation on the LayerNorm layers in the URMA architecture.
Figure shows that removing the LayerNorm layers leads to a significantly slower learning process and lower final performance.
E Ablation on Two Joint Description Encoders
In the URMA architecture, we use two separate encoders for encoding the joint description vectors d, for the joint observation encoding and the universal action decoding. the joint encoder, use the fo MLP and in the universal decoder, two separate encoders, are stopped, so fø is optimized for both the joint encoder and the universal decoder. We test the two cases where the representation used for the universal decoder is either taken after the softmax operation (gw = full fạ) or before the softmax, i.. after the tanh operation (gw partial fø).
the Iw MLP. To ablate the necessity of having replace Iw with fo in the universal decoder. None of the gradients
Training all 16 robots
50
Average return
40
30
10
0.0
0.2
0.4
0.6
0.8
1.0
Environment steps per robot
1eB
URMA:
URMA: g.. partial
URMA: full
Figure 5: Ablation on the usage of two separate or one single joint description encoder the joint encoder and the universal decoder.
Figure 5 shows that using the same joint description encoder in both places leads to slightly worse performance when using the full fo representation and the same performance when using the partial fo representation. Therefore, having two separate encoders for the joint descriptions does not seem be necessary. However, might add more flexibility, allowing the policy to learn two sets of encodings when needed.
F
Ablation on Robot Mass and Dimensions in Observation Space
URMA uses the robot mass and dimensions L, W and H in the general observations og and as part of the joint and feet descriptions dy and d,. To make sure that the policy does not overfit to
15

---

specific values of these attributes, we use strong randomization of the mass of the torso and slight randomization of joint nominal positions, which leads to slight changes in the robot bounding boxes (i.e. robot dimensions). To evaluate the necessity of even having the mass and dimensions in the observations, we remove them from the observations and descriptions.
Training on all 16 robots
50
Average retum
40
30
20
10
0.0
0.2
0.4
0.6
0.8
1.0
Environment steps per robot
1eB
URMA
URMA without m, L, W, H
Figure 6: Ablation on the mass and robot dimensions in the observations and descriptions.
Figure 6 shows that removing the mass and dimensions from the observations and descriptions leads to the same performance. Therefore, the policy does not require the robot's 's mass and dimensions to learn strong locomotion for the different robots.
G Ablation on Single Set of Reward Coefficients for All Robots
Getting the most fine-tuned gait for every robot requires tuning the reward coefficients for all robots individually. Mainly, the relative weighting between the two command tracking terms and all the penalty terms needs to be adjusted. More inherently unstable platforms like humanoids need lower penalties at the start of training, to avoid the policy getting stuck in local minima when trying to stabilize and comply with the penalties at the same time. Furthermore, specific joint properties, e.g. joint accelerations or joint torques, need to take on larger values for bigger and heavier robots and also the penalties based on their squared norms (Table 2) scale with the number of joints of the robot. Nethertheless, only tune 7 out of the 14 reward coefficients for all robots (Table 3), where many of these coefficients are set to the same similar values for robots of the same morphology. The precise tuning of reward coefficients is not necessary to learn robust policies with good locomotion gaits, rather it is an engineering tool to get fine-grained control over the policy's final behavior.
However, possible to use the same set of reward coefficients. To demonstrate this, we perform an ablation study where we train an URMA policy with the same set of reward coefficients for all robots and the same reward curriculum length (URMA: Single reward set) and compare it to policy with our differently tuned reward coefficients (URMA: Multi reward sets). We evaluate the single reward set policy on all robots using the tuned reward coefficients (URMA: Single reward set, eval multi reward sets) for fair comparison. We chose the reward coefficients and the reward curriculum length for the single reward set policy to be conservative (Table 5), i.c., we use high tracking coefficients and low penalty coefficients, to ensure that every robot will learn to walk.
Robot
T1
T2
T3
T4
T5
T6 T7 T8
T9
T10 T11 T12
T13 T14 T
All robots 5.0
2.5
2.0
0.05
0.2 0.2
10.0 2.5c-7
2c-5 6e-3
30.0
1.0
0.1 0.5 80e6
Table 5: Reward coefficients Te for all the reward terms for all robots. The last column T is the time-dependent reward curriculum length until all penalties are linearly increased from zero to their final values. The description of the reward terms can be found in Table
Figure shows that using the same set of reward coefficients for all robots keeps the stable learning process and its final performance the differently tuned reward coefficients is only slightly worse
16

---

Training all robots
Average return
40
20
0.0
0.2
0.4
0,6
0.8
1.0
Environment steps per robot
1e8
URMA: Multi reward sets
URMA: Single reward set
URMA: Single reward set, eval multi reward sets
Figure 7: Ablation on using the same set of reward coefficients for all robot embodiments.
than the policy trained directly with the tuned reward coefficients. Therefore, using the same set of reward coefficients for all robots is valid option to simplify the need for engineering going into the training process and still achieve good locomotion gaits.
H Ablation on Minimal Quadruped Set for Zero-Shot Transfer
To evaluate the impact of the number of robots and specifically quadrupeds in the training set re- quired for zero-shot transfer new quadruped, we perform an ablation study find a minimal set of quadrupeds that are needed achieve good zero-shot transfer performance the Unitree A1. As the Unitree Al is most similar to the Unitree Gol and Go2 robots, we train policy only on those two robots and test if they are already enough to transfer to the Al. Further, we add the Barkour vO (Bkv0), Barkour vB (Bkvb) and the MAB Robotics Silver Badger (SBg) to the training set to see the performance improves with these additional robots. Then, we train on all quadrupeds besides the A1, i.e. including the ANYmal and C and the Petoi Bittle. Finally, we train on the full set of robots besides the AI, so including also the biped, humanoids and hexapod robots.
Zero-shot on Unitree A1
Average return
30
20
10
0.0
0.2
0.4
0.6
0.8
1.0
Environment steps per robot
1eB
URMA: All robots but A1
URMA: All quadrupeds but
URMA: Gol, Go2, Blvo, Blevb, SBg
URMA: Go1, Go2
Figure 8: Ablation the minimal set of quadrupeds needed for zero-shot transfer the Unitree A1 robot.
Figure shows that training on the Unitree Gol and Go2 robots already leads to some decent zero- shot transfer performance on the A1. Including the Barkour v0, Barkour vB and the MAB Silver Badger improves the performance further. Training on quadrupeds besides the Al leads to the best zero-shot transfer performance and on par with training on all robots. Therefore, having strongly similar robots the training set (Gol and Go2) is enough get some initial zero-shot transfer performance, but using the most diverse set of quadrupeds leads to the best transferability in the end.
I Ablation on Humanoids for Zero-Shot Transfer and Fine- Fine-Tuning
We evaluate the zero-shot and fine-tuning performance of URMA on the Unitree HI and the PAL Robotics Talos humanoid robots by following the same procedure for the MAB Robotics Silver
17

---

Badger quadruped (Figure 3 bottom left). We first train the policy on all robots besides the HI / Talos, while evaluating the zero-shot performance on the H1 / Talos, and then fine-tune the final policy on the / /aalos and evaluate the performance again. A standard URMA policy trained on all robots and one trained on only the specific robot from scratch are used as a baseline for comparison.
Zero-shot and fine-tuning on Unitree H1
40
Average return
30
20
10
0.0
0.2
0.4
0.6
0.8
1.0
1.2
Environment steps per
1eB
URMA: 15 robots
URMA: robots, fine-tuning H1
URMA: Only H1
URMA: robots
Figure 9: Ablation the zero-shot and fine-tuning performance of URMA the Unitree H1
humanoid robot when pre- pre-training on all other 15 robots.
Zero-sher shot and fine-tuning PAL Robotics Talos
Average retur
60
40
20
0.0
0.2
0.4
0.6
0.8
1.0
1.2
Environment steps per robot
URMA: 15 robots
URMA: robots, fine-tuning Talos
URMA: Only Talos
URMA: 16 robots
Figure 10: Ablation on the zero-shot and fine-tuning performance of URMA on the PAL Robotics Talos humanoid robot when pre-training on all other 15 robots.
Generalization across our set of biped and humanoid robots is challenging due to the small set of highly variable embodiments and the inherent instability of the humanoid form factor. Figure 9 and Figure 10 show this difficulty, as the policy is able to do any initial transfer the Unitree Hi the Talos. We hypothesize that more biped and humanoid robots in the training or additional robot randomization on the current robots necessary for the policy to get some meaningful zero- shot transfer performance. Nethertheless, URMA learns strong general locomotion features during the initial training on other robots, including other humanoids, and is able to quickly adapt to the HI / Talos during fine-tumin une-tuning. The fine-tuned policy is able to learn quicker and achieve higher performance in the 20M steps of fine-tuning than the multi multi-emmboiment and single-robot baseline policies trained from scratch in the same short amount of time. While the fine-tuning policy and the policies trained from scratch will reach the same performance in the end, fine-tuning with already embodiment.
good representation space allows for quick adaption and fast learning
Ablation Inter-Morphology Generalization
Having different robot morphologies in the training set can potentially hurt or help the performance on robots of specific morphology, leading to inter-morphologg generalization. We evaluate the impact of training on additional morphologies by comparing the performance for either quadruped or biped and humanoid robots of a policy trained on all robots with a policy trained only on the respective morphology.
Figure 11 shows that training additional morphologies does not hurt the performance in the of quadruped robots and marginally decreases the performance in the case of biped and humanoid
18

---

Performance quadrupeds
Performance all bipeds & humanolds
40
Average return
Average retum
60
40
20
20
10
0.0
0.2
0.4
0.6
0.8
0.0
0:2
0.4
0.6
1.0
Enviranment steps per robot
Environment steps per robot
URMA Training with all 16 robots
URMA: Training with all quadrupeds
URM/: Training with all 6 bipeds humanoids
Figure 11: Ablation on the impact of training on additional morphologies when considering only quadruped (Left) or biped and humanoid (Right) robots.
robots. We test the performance on robots that are already in the training set, so is reasonable that the policy can perform best when training only on these robots. In the quadruped case, the policy seems have enough capacity also fully fit the humanoids and hexapod without losing perfor- mance. We think that nntrr-morpholoyy generalization and reasoning about multiple morphologies can still occur and help the policy in the zero-shot and few-shot case, especially when encountering embodiments that mix properties of previously seen robots from different morphologies.
K Investigation on Policy Conditioning on Joint Descriptions
The joint description vectors dj are integral part in URMA. In the joint encoder, they used to route the corresponding joint observations to the correct dimension in the joint latent vector Bjoints- In the universal decoder, the joint descriptions are used to unpack the joint specific actions from the action latent vector Aaction To investigate how strongly the policy conditions on the joint descriptions, we first completely shuffle the joint descriptions for an already trained policy and test it on the Unitree Al quadruped and the Unitree H1 humanoid. The A1 is able to stand an zero velocity command but when receiving any non-zero velocity command, struggles and falls after one two steps. The HI is not able to stand at all and falls mostly immediately, due to the stronger inherent instability of the humanoid morphology. The behavior in both cases does not appear random but rather systematic and strongly uncoordinated between the joints, This makes sense as the policy learned smooth actions but the correct mapping is mixed up.
We also test the conditioning of the policy on different attributes of the joint descriptions. We take the Unitree A1 and strongly perturb the joint descriptions (multiplying the associated values by either 1/3 or 3) for the PD gains + action scaling factor, the joints rotor inertias, and the joints friction coefficients. When we increase the values of the description of the PD gains + action scaling factor, the policy assumes the robot's 's motors and controller are stronger and more responsive than they are. The same situation occurs for the rotor inertia and friction coefficient, the policy assumes that the joints are less responsive and smooth, causing performance degradation.
PD gains action scaling factor
Joint rotor Inertia
Joint friction coefficient
10.5
10.5
10.5
Average action norm
Average action norm
Average action norm
10.0
9.5
9.5
9.0
9.0
9.0
8.5
8.5
8.5
1/3
1/3
1/3
Attribute factor
Attribute factor
Attribute factor
Figure 12: Multiplying the value for attributes of the joint descriptions for the Unitree A1 with different factors. Left Varying the value for the gains action scaling factor. Middle Varying the value for the joints rotor inertias. Right- Varying the value for the joints friction coefficients.
19

---

The left side of Figure 12 shows that decreasing the description values for the PD gains and action scaling factor by a factor of 1 /3 leads to the policy trying to overcompensate compared to the normal values (factor 1) by producing actions with a higher average magnitude measured over 10 episodes. In the opposite case, increasing the values by factor with a lower average magnitude, as it thinks the robot is more responsive than it actually is. A similar but less pronounced effect can be seen in the middle and right side of Figure 12 for the rotor inertias and friction coefficients. Lower perceived inertias and friction coefficients in the joints (multiplying the values for every joint with 1/3) leads to the policy producing actions with lower average magnitude and vice versa. In general, the policy seems to be able to capture the changes in the different attributes of the joint descriptions and adapt its actions to their physical implications.
leads to the policy producing actions
L Individual Robot Returns
ANYbotics AWimmll B
ANYbotics ANimall C
Google Barkour vp
20
0.0
0.2
0.5
0.0
0.4
0.4
1.0
0.9
0.2
0.6
Google Bariour vB
MAB Robotics Silver Barger
Petol Bittle
Aueraga ratum
15 15
0.0
022
0.4
0.5
0.5
1.0
0.0
244
0.5
0.5
1.0
0.0
0.2
0.4
0.5
0.5
Unitree Al
Unitree Gol
Unitres Go2
Avo agua paum
2
06
0.0
0.4
64
0.0
0.9
Agility Robotics Cassie
PNL
Robotts OPS
202
Aoerage raturn
0.0
0.2
0.4
U.
045
0.0
02
2.4
3.5
0.3
1.0
0.0
022
Us
Softbard Robaties NAG VS
Untitree G1
Unitrree H1
Avuraga retum
25
0.0
52
2.4
0.5
100
0.0
022
5.4
0.5
54
3.0
0.0
022
5.4
0.5
03
10
per
Custom Haxapod
Aeusage noaur
15
0.0
5.4
0.
3.0
Enviromment steps
URMA
Multi-Head Baseline
Padding Baseline
Single-robot training
Single-robot training at 1.6B steps
Figure 13: Individual returns for all robots. URMA, the multi-hear and the padding baseline are trained on 16 robots together. The single-robot policy is trained on only one robot. It needs more samples per robot to catch up with the multi-embodiment training. The empirical maximum performance when continuing the training for 1.6B steps on a single robot shown as dashed line.
20

---

M Extended Theoretical Analysis and Proofs
This section provides detailed proof of Theorem 1.
First, we show that the loss function in (4) is normalized and bounded [o, 1]. The PPO loss optimized by gradient descent-for single pair Xmi, Ymi is defined as follows:
x(als, d) Ta(a s,
A", clip( T(als,d) mo(a|s.d)
1- e, +1 3)
[(u(th(((Xms)) Ymc) = min
where 8, d and the observations, descriptions and actions coming from the sample X ([aas) mo(h(((Xme))) ro d) is the fixed probability density of the first PPO iteration from Ym and AA is the true advantage under from Yi To bound L. we need to bound Ae and the ratio x(als.d) in (9).
mo(a|a,))
By looking the definition of the advantage A Q V",
bound it between Amin Rais Rus and Amax Buss Ruis, where Rmin and Rmax are the minimum and maximum possible I-y I-7'
I-y I-7
reward. In our training environments, Raiin and mmx is the reward given when the policy tracks Rass
Ram and Amax 1-7' the given target commands perfectly. Therefore, in our setting Amin I-7
To bound in (9), we need to look at the following four cases:
If A and r < 1 then is bounded by 1 e.
If AA > 0 and r > then is bounded by 1 + e.
If A 0 and r < 1 then is bounded by 1
If A and > 1 then is not directly bounded, but modify the PPO training to discard (s, d, a) pairs with too big probability ratios, i.c. only allowing samples with < 1 + E where E is some upper probability ratio that we allow in the case of negative AT.
With the bounds on AT and r, we can normalize and bound the loss function to be in [0, 1] by defining as
((((h(((mm)))) Yms) ((W(h(((Xmm))) Ymns) Imin
(10)
1max
Iiin
where 4min = Am (1
Buss (1 e) and mmax Amia (1 + = I-ry
Busa (1 E). I-
Because our encoder and decoder w functions take sequences of joint and feet observations and descriptions as input, we rely on the attention mechanism in the encoder and our universal morphol- ogy decoder to flatten their respective outputs for the following functions. In practice, in the encoder, sum over zj get rid of the sequence dimension (see (2)), and at the end of the decoder, concatenate the resulting actions for every joint of a robot to get the flat vector = [a a² a a').
To ensure every vector of the same length, we pad them with zeros.
We start the proof of Theorem by bounding the excess risk, i.e. the difference between the risk (fve(f, h w) where f. h, v are the minimizes of the empirical risk, and the minimum task averaged- risk E:gg Notice that excess risk represents how much pay in terms of performance w.r.t. the optimal policy update since are optimizing policy using only finite amount of samples. We
21

---

decompose the excess risk, as done in D'Eramo et al. [26], as follows:
Eavg h, u)) Eing
Eang(f, (ff,hh,u
Eavg
n.M Σ
((m(h(((Xm))))) Mmi)
+
((x(h(((X Yme)
Σ
E(wi (I" (Xmi))),, Yin))
n.M
nM
+
Σ TAI
E(w m)), mms) Eavg
(11)
n.M
To bound the complete difference, will bound its three components A, B and C. First, B is bounded by O, as f. h and w are the true minimizers of the empirical risk and therefore f*, ht and w" can not achieve lower loss on the empirical dataset. Next, C' can be bounded with Hoeffding's inequality with probability 1 8
2 by [I(((aa) (2nM) as it contains only nM random variables bounded
the interval [0, 1]. Finally, to bound term A, we begin with defining the following auxiliary sets, which make the rest the derivations cleaner:
S = L(W(H(F(X))) Y) = ((((w(((((Xm))) Km.)) W.W.EE,,
C MR
S' W(H(F(())) {(xo(h(((Xmm)) Mm))) w e, S RM
S" =H(((X) {(b(f(Xms)) Yms) H. f F} S M
Mn x X
We start to bound A with probability 5 using Theorem from Maurer et al. [41]:
Eag (f,h, f. a))
Σ
nM
1
(avg(f, h,,), nM mi
Σ [(((((((Xmi))) Yms)
sup JEF.NEH.WEE
V2TG(S)
9 (2)
(12)
nM
2nM
With the contraction lemma, Corollary 11 of Maurer et al. [41]. we are bounding G(S) < (())G(S'). Compared Maurer al. [41], we can't guarantee that the PPO loss is 1-Lipschitz, and therefore, keep the Lipschitz constant of L(E). In practice, this is no problem, it is just bound the resulting Gaussian complexity of S with Theorem 12 from Maurer et al. [41], given the universal constants ci and dg as
constant that can be reduced by scaling the loss. Furthermore,
G(S') S (TTWWG((S))+ +
+ min G(W(y)), yEY
(13)
where T.(W) is the largest value for the Lipschitz constants in the decoder function space W, and D(S") is the Euclidean diameter of the set S". We finish the bounding of the Gaussian complexities of the auxiliary sets by using Theorem 12 from Maurer et al. [41] once more. For the universal constants el' and dy we bound
(UGSG4444HGGGF + min G(H(p)).
(14)
pEP
22

---

Combining the bounds on G(S") and G(S"), we have
G(S') (
+ + min G(H(p)))
PEp
+GD(("")(W)) + min G(W(U)) NEY
min G(H(p)) PEP
+cDD(S")(W) + min G(W())).
(15)
yEY
We continue to bound the Euclidean diameters in (15) with D(S") I 2 2su (X)([ and D(F(X)) < /x To minimize the last term of (15), we define do = 0 and w(2, do. a) = 0, Ve E W. where is the resulting representation coming from h and o, d and a are the observations, descriptions and actions coming from the sample Xmi resulting in minyey G(W(y)) = G(W(O)) = We finish the bounds on Gaussian complexities of the auxil- iary sets by combining them:
G(S) (L0)GG((LLLLLLHH)(EX))) 2c, supl[((X)I((())
JEF
+
LLWWW min G(H(P)) + 2ch sup (l( (X
(16)
pEp
h.f
We finalize the bound on term A from (11) by substituting (16) in (12):
aavg (f, w))
1
Σ
(()((((())))) Ymi)
nM
22 L(E) +LCLIW))LCHHHGG(TZ)))+
sxp]I((X)((((H) Fet
nM
+ L(W(W min G(H(P)) + 2dh supplb(f(x))/0mm)) + PEP
h.f
2nM
(()I(I)(())((((X) L()) sup flf
nM
nM
((W) minper G(H(p)) + sup,, (I(((())) 9 3
c3
nM
nM
2nM
To complete the proof, we compute the union bound between the three components A, B, and C of (11). Remembering that B always non-positive, the bound for A holds with probability 6, and the bound for C' holds with probability 1 - 1. we assume that the probability of violation of both bounds will be 8' 8 8 35. With everything in place, we finally bound the overall excess risk
Eag (f. h, ()) - Elng Sc ((OG(E)
nM 100sup;)1/(())]]144)000
+ C2
nM [(()L(W minpeP G(H(p))
+ c3
nM
I()) (
+
nM
8ln(2) V nM
+
where we have renamed 8' as 6.
23


Reviews:
[{"review_id": "F65Bu5EDPd", "reviewer": "Reviewer_bzcE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes the Unified Robot Morphology Architecture (URMA) to learn a single locomotion policy for robots with different morphologies. This study uses the Multi-Task Reinforcement Learning (MTRL) concept to share knowledge between tasks and learn a common representation space applicable to all tasks. In the proposed method, embodiment-specific observations are encoded and combined into a single joint latent vector to handle the varying sizes of observations for each robot. To determine the final action for the robot, a universal morphology decoder is employed. This decoder takes the general action latent vector, pairs it with the set of encoded specific joint descriptions, and combines it with the single joint latent vectors to produce the mean and standard deviation of the actions for every joint. Overall, the proposed idea is intriguing, and targeting the challenge of cross-embodiment learning is a nice effort. However, the paper's presentation, comparison, and quantification need significant improvement.", "summary_of_recommendation": "After considering the rebuttal, I have updated my decision to \"weak accept.\"", "technical_quality": 2, "clarity_of_presentation": 2, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "Please review the questions mentioned in the review section and check the comments.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vWzR6dXwLI", "reviewer": "Reviewer_kaLC", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a method for learning a general locomotion policy that can be used on multiple diverse robot platforms, varying from quadrupeds to humanoids to hexapods. This abstract controller can then be zero-shot transferred to unseen platforms of similar nature to those seen in training, i.e. a different unseen quadruped robot. By encoding the information of each robot joint individually and then combining them with an attention head into a latent vector, networks of the same size can be used regardless of the number of joints each robot has. The authors then train their system with 16 different robots and show that it can zero-shot transfer to unseen platforms that are similar nature and can be quickly fine-tuned if the morphology differs a lot.", "summary_of_recommendation": "A good step towards the possibility of an actual \"robot foundation model\". Novel control architecture that can scale and handle robots with arbitrary number of joints/actuators. Shows successful zero-shot transfer and fine-tuning capabilities in limited settings with possibility for more generalization.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Is there no MLP encoding for the feet observations/descriptions? (There is no MLP blocks there in Figure 2, they go straight into the attention encoding)\n\nWhy is there a different joint description encoding used to produce the mean and std dev of the action? Why can’t the same $f_\\phi$ encoding be used?\n\nWhy do the single-robot training baselines do so poorly on the quadruped scenarios? You mention that this is because of aggressive DR in order to facilitate sim2real transfer, but from the learning curves it looks like they won’t learn at all, not just that they require more samples (the learning curve seems to have plateaued already). Is this the case or if given more samples will the single-robot baselines actually learn? If not then that is very strange since there are many multiple prior works (that you reference, [1-10]) that can solve the quadruped locomotion task.\n\nCan you provide ranges for the DR used, especially since there seems to be some odd choices of things to randomize? Why would feet size and torque and velocity limits need to be randomized, these should all be pretty well known and fixed right? What is the “action scaling factor”, is this literally some scaling of the policy action? What is the point of that, even if the resulting torque is different on hardware, we know absolutely for certain what action we are trying to command right? What is “motor strength”, is this like some torque scaling factor? Would the randomization of the PD gains not have the same effect as this?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6lArZO3mzo", "reviewer": "Reviewer_xWHF", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a archetecture to train a unified policy that can control many different legged robots.", "summary_of_recommendation": "I think more analysis/evidence for the benefit of using a unified policy is needed.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "(1) I don't understand the theoretical analysis part of the paper. What property of the URMA is used in the analysis that other baseline methods do not have? It would be great to explicitly state things like: \"Our URMA has this property XXX, and using the property, we can derive this bound (without this property the bound would not work). While other architectures lack this property.\"\n\n(2) The single robot training baseline performs very badly. Most papers in RL for quadruped use single robot training and they can demonstrate pretty good performance. I wonder where is this gap coming from.\n\n(3) What do the motions of the bipedal robots look like? How important is it to include them for training if all I care about is quadruped robots? How about fine-tuning performance on bipedal robots?\n\n(4) For the quadruped robot with a spine, is the policy fine-tuned to utilize the spine in any way? e.g., does it discover a strategy that quadruped with no spine cannot do?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Post-rebuttal metareview**:

This paper proposes a novel locomotion policy architecture, URMA (Unified Robot Morphology Architecture), which learns a single policy for different types of legged robots. The cross-embodiment capability and the proposed architecture are pretty novel. Most of the reviewers' concerns are addressed in the rebuttal. Adding more real-world results for non-quadrupedal robots will further strengthen the paper.

----------------
**Pre-rebuttal metareview**:

Strengths:
1. The proposed idea to solve cross-embodiment locomotion is novel and interesting.
2. Real-world experiments on quadrupeds are promising.

Weakness:
1. Presentation needs improvement.
2. Need more fair and comprehensive comparison. Why is the specialized policy worse than the main policy?
3. No real-world experiments for non-quadruped robots, which is the main point of the paper.
4. The “first step towards foundation model” argument needs justification.
5. Other issues mentioned by reviewers such as presentation, training setups, lack of ablation, etc.
Decision: Accept