One Policy to Run Them All: an End-to-end Learning Approach to Multi-Embodiment Locomotion

Authors: nico  bohlinger
grzegorz  czechmanowski
maciej  krupka
piotr  kicki
krzysztof  walas
jan  peters
davide  tateo
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 00:31:01

Abstract: Deep Reinforcement Learning techniques are achieving state-of-the-art results in robust legged locomotion.
While there exists a wide variety of legged platforms such as quadruped, humanoids, and hexapods, the field is still missing a single learning framework that can control all these different embodiments easily and effectively and possibly transfer, zero or few-shot, to unseen robot embodiments.
To close this gap, we introduce URMA, the Unified Robot Morphology Architecture. Our framework brings the end-to-end Multi-Task Reinforcement Learning approach to the realm of legged robots, enabling the learned policy to control any type of robot morphology.
The key idea of our method is to allow the network to learn an abstract locomotion controller that can be seamlessly shared between embodiments thanks to our morphology-agnostic encoders and decoders. This flexible architecture can be seen as a first step in building a foundation model for legged robot locomotion.
Our experiments show that URMA can learn a locomotion policy on multiple embodiments that can be easily transferred to unseen robot platforms in simulation and the real world.

Main paper: 
 introduction: 
 The robotics community has mastered the problem of robust gait generation in the last few years. With the help of Deep Reinforcement Learning (DRL) techniques, legged robots can show impressive locomotion skills. There are numerous examples of highly agile locomotion with quadrupedal robots [1, 2, 3, 4, 5, 6] , learning to run at high speeds, jumping over obstacles, walking on rough terrain, performing handstands, and completing parkour courses. Achieving these agile movements is often enabled by training in many parallelized simulation environments and using carefully tuned or automatic curricula on the task difficulty [7, 8] . Even learning simple locomotion behaviors directly on real robots is possible but requires far more efficient learning approaches [9, 10] . Similar methods have been applied to generate robust walking gaits for bipedal and humanoid robots [11, 12, 13] . The learned policies can be effectively transferred to the real world and work in all kinds of terrain with the help of extensive Domain Randomization (DR) [14, 15] during training. Additionally, techniques like student-teacher learning [1, 16] or the addition of model-based components [17, 18] or constrains [19, 20, 21] to the learning process can further improve the learning efficiency and robustness of the policies.At the same time, new advances in computational power, the availability of large datasets, and the development of foundation models are opening new frontiers for artificial intelligence, allowing us to implement and learn more complex and intelligent agent behaviors. Future robots will require incorporating these models into the control pipeline [22, 23] . However, to fully benefit from foundation models, we need to be able to integrate these high-level policies with the low-level control of the robots. The long-term objective would be to develop foundation models for locomotion, allowing zero-shot (or few-shot) deployment to any arbitrary platform. However, to reach this objective, it is fundamental to adapt the underlying learning system to support different tasks and morphologies. Therefore, we argue that the Multi-Task Reinforcement Learning (MTRL) problem is a fundamental topic for the future research of robot locomotion, and, indeed, this formulation has recently attracted the interest of the community, using both structured [24] and end-to-end learning approaches [25] . MTRL algorithms share knowledge between tasks and learn a common representation space that can be used to solve all of them [26, 27] . To map differently sized observation and action spaces into and out of the shared representation space, implementations often resort to padding the observations and actions with zeros to fit a maximum length [28] or to using a separate neural network head for each task [26] . These methods allow for efficient training but can be limiting when trying to transfer to new tasks or environments: for every new robot, the training process has to be repeated from scratch, as different embodiments require different hyperparameters, reward coefficients, training curricula, etc. Already in the case of the same robot morphology, e.g. quadrupeds, a trained policy can not be easily transferred when the number of joints is not the same for the robots. This is even more evident when trying to reuse the learned gait across different types of morphologies. This issue is closely related to the fundamental correspondence problem in robotics [29] , as the policy has to learn an internal mapping between the different action and observation spaces and the embodiments themselves, which define the robots' kinematics. In practice, the number of joints and feet of a legged robot determines the size of its action and observation space, which can differ for every new robot. This often prevents a straightforward transfer of existing policies as the learning architecture fully depends on the specific robot platform.To tackle this problem and to move to more powerful and general policies that can be used as locomotion foundation models, we propose a novel MTRL framework that allows simultaneously learning locomotion tasks with many different morphologies easily and effectively. Our approach is based on a novel neural network architecture that can handle differently sized action and observation spaces, allowing the policy to adapt easily to diverse robot morphologies. Furthermore, our method allows us zero-shot deployment of the policy to unseen robots and few-shot fine-tuning on novel target platforms. We highlight the effectiveness of our approach, first with a theoretical analysis and then by training a single locomotion policy on 16 robots, including quadrupeds, hexapods, bipeds, and humanoids. Finally, we zero-shot transfer the learned policy to two simulated and three realworld robots, showing the transferability and robustness of our method.Early work on controlling different robot morphologies is based on the idea of using Graph Neural Networks (GNNs) to capture the morphological structure of the robots [30, 31, 32] . Each node in the graph represents a joint of a robot, and its state is comprised of the joint's specific information, e.g. current position, velocity, etc. Through message passing, the GNN can then aggregate information from neighboring joints and learn to control the robot as a whole. GNN-based approaches can control different robots even when removing some of their limbs, but they struggle to generalize to many different morphologies at once, as every morphology requires a different graph structure. Furthermore, the local nature of message passing can lead to information bottlenecks in the policies and the inability to act as a cohesive global controller [33] .Transformer-based architectures have been proposed to overcome the limitations of GNNs by using the attention mechanism to globally aggregate information of varying numbers of joints [33, 25, 34, 35] . These methods still lack substantial generality as they are limited to morphologies that were defined a priori. For example, Kurin et al. [33] uses encoder-decoder pairs for each type of joint, which limits the architecture to a set of predefined joint types and does not allow for components that only have associated observations but no actions, e.g. the feet of legged robots. Trabucco et al. [25] defines tokens for each type of observation and joint, which is a more general system, but those tokens are handcrafted, and there is a different set of joint tokens for every morphology, which makes generalization between morphologies and to new ones difficult or even impossible.Besides the GNN-and Transformer-based methods, which consider only environments with 2Dplanar or physically implausible robots, Shafiee et al. [24] recently showed that a single controller can be trained to control 16 different 3D-simulated quadrupedal robots and to transfer to two of them in the real world. Their method uses a Central Pattern Generator (CPG) and inverse kinematics to generate and track trajectories of the four feet of the robots. This approach has the drawback of discarding the joint-specific information, and the controller can be deployed only on robots with the same number of feet. Furthermore, Feng et al. [36] and Luo et al. [37] use procedurally generated quadrupeds in simulation to train a single policy that transfers to unseen real-world quadrupeds. However, their learning framework assumes that every robot is in the same morphology class, and their neural network architecture can only deal with robots with the same number of joints. Compared to all the other approaches, our method can handle multiple embodiments from any legged morphology, can adapt to arbitrary joint configurations with the same network and can be deployed on real world robots. 
multi-embodiment locomotion with a single policy: 
 In MTRL the objective is to learn a single policy π θ that optimizes the average of the expected discounted return J m (θ) over the reward function r m across M tasks:EQUATIONwhere τ is a trajectory given by the state-action pairs (s t , a t ), γ is the discount factor, and T is the time horizon. In our case, we consider different robot embodiments as separate tasks and train a policy controlling all robots and optimizing the objective described in (1) . We aim to design a policy where the underlying neural network architecture is independent of the set of possible embodiments. Therefore, we propose the Unified Robot Morphology Architecture (URMA), which is completely morphology agnostic, i.e. it can be applied to any type of robot with any number of joints such that there is no need to define the possible morphologies or joints beforehand. We use URMA to learn robust locomotion policies, but its formulation is general enough to be applied to any control task. Figure 2 presents a schematic overview of URMA. In general, URMA splits the observations of a robot into distinct parts, encodes them with a simple attention encoder [38] with a learnable temperature [39] , and uses our universal morphology decoder to obtain the actions for every joint of the robot. F e e t Attention encoding . Figure 2 : Overview of URMA. Left -Joint observations and descriptions are encoded and combined into a single joint latent vector through an attention head. Bottom center -Feet observations and descriptions are encoded in the same way. Top center -Joint latent, feet latent, and general observations are fed through the core network to get the action latent vector. Right -The universal morphology decoder encodes the joint descriptions and pairs them with the action latent vector and the single joint latent vector to produce the action mean and standard deviation for the final action.To handle observations of any morphology, URMA first splits the observation vector o into robotspecific and general observations o g , where the former can be of varying size, and the latter has a fixed dimensionality. For locomotion, we subdivide the robot-specific observations into joint and feet-specific observations. This split is not necessary but makes the application to locomotion cleaner. In the following text, we describe everything w.r.t. the joint-specific observations, but the same applies to the feet-specific ones as well. Every joint of a robot is composed of joint-specific observations o j and a description vector d j , which is a fixed-size vector that can uniquely describe the joint by using characteristic properties like the joint's rotation axis, its relative position in the robot, torque and velocity limits, control range, etc. (see Appendix A for more details). The description vectors and joint-specific observations are encoded separately by the Multilayer Perceptrons (MLPs) f ϕ and f ψ and are then passed through a simple attention head, with a learnable temperature τ and a minimum temperature ϵ, to get a single latent vectorEQUATIONthat contains the information of the joint-specific observations of all joints. With the help of the attention mechanism, the network can learn to separate the relevant joint information and precisely route it into the specific dimensions of the latent vector by reducing the temperature τ of the softmax close to zero. The joint latent vector zjoints is then concatenated with the feet latent vector zfeet and the general observations o g and passed to the policies core MLP h θ to get the action latent vector zaction = h θ (o g , zjoints , zfeet ). To obtain the final action for the robot, we use our universal morphology decoder, which takes the general action latent vector and pairs it with the set of encoded specific joint descriptions and the single joint latent vectors to produce the mean and standard deviation of the actions for every joint, from which the final action is sampled asEQUATIONTo ensure that only fully normalized and well-behaved observations come into the network, we use LayerNorm [40] after every input layer. The learning process also benefits from adding another LayerNorm in the action mean network µ ν . We argue that this choice improves the alignment of the different latent vectors entering into µ ν better. To ensure a fair comparison, we also use LayerNorms with the same rationale in the baseline architectures. We highlight the benefits in the ablation study on the usage of LayerNorm for URMA in Figure 4 of Appendix D.Our second contribution is the open-source modular learning framework, which enables us to easily train robust and transferable locomotion policies for all kinds of legged robots. When adding a new robot to the training set, only the reward coefficients, controller gains, and domain randomization ranges have to be adjusted, which can be easily done by slightly modifying the ones from existing robots in the framework. As the penalty terms in the reward function are not essential for learning the core locomotion but only shape the resulting gait, we apply a time-dependent fixed-length curriculum r c (t) = min t T , 1 r T c , where t is the current training step, T is the curriculum length, and r T c is the final penalty coefficient. This speeds up the learning and makes the coefficient tuning process easier and more forgiving, as the policy can handle higher penalties better when it has already learned to perform basic locomotion. The full details on the environment setup, reward function, and domain randomization can be found in Appendix A.To evaluate the benefit of learning shared representations across robot morphologies through URMA, we extend the multi-task risk bounds from Maurer et al. [41] and D'Eramo et al. [26] to our morphology-agnostic encoder and decoder. As we will use the Proximal Policy Optimization (PPO) algorithm in our learning framework, we frame our problem as an evaluation of the performance difference between the PPO training on the empirical dataset and the optimal policy update with infinite samples. In our simplified analysis, we will assume that i. our policy optimization step can find the policy that minimizes the surrogate loss over the current dataset; ii. the trust region is small enough such that the surrogate loss and the expected discounted return of the policy are close enough; iii. the surrogate loss is computed with the true advantage. Given our set of robot embodiments µ = (µ 1 , . . . , µ M ), the set X ∈ X M n of n input samples from the space of observations, descriptions and actions X for each of the M tasks and the corresponding set Ȳ ∈ R M n of advantages A π divided by the initial probabilities π 0 , we can describe the policy learning as finding the minimizer f , ĥ, ŵ of the following optimization problem:EQUATIONwith f ∈ F : X → R J × X , h ∈ H : R J × X → R K × X and w ∈ W : R K × X → R being the encoder, core and decoder networks, and ℓ : R × R → [0, 1] the normalized policy optimization loss function. We quantify the performance of the functions f, h, w with the task-averaged riskEQUATIONWe define ε * avg as the minimum of this risk, with the minimizers f * , h * and w * . We measure the complexity of some function class Z composed of K functions via the setEQUATIONwhere γ mki are i.i.d. standard Gaussian random variables. Furthermore, we define L(Z), as the upper bound of the Lipschitz constant of all z in Z, and the Gaussian average of Lipschitz quotientsEQUATION)where γ is a vector of d i.i.d. standard Gaussian random variables and z ∈ Z : Y → R d with Y ⊆ R p . Using the definitions above, we can bound the risk ε avg : Theorem 1. Let µ, F, H and W be defined as above and assume 0 ∈ H and w(0) = 0, ∀w ∈ W.Then for δ > 0 with probability at least 1 -δ in the draw of ( X, Ȳ) ∼ M m=1 µ n m we have thatEQUATIONThe proof of Theorem 1 and a discussion on the assumptions can be found in Appendix M. For reasonable function classes W, the Gaussian average of Lipschitz quotients O(W) can be bounded independently from the number of samples. For most settings, the Gaussian complexityG(F( X)) is O( √ nM ). Also the terms sup f ∥f ( X)∥ and sup h,f ∥h(f ( X))∥ are O( √ nM ), if they can be uniformly bounded. Using these assumptions, the URMA policy structure is better suited for multitask learning as all the first four terms are O ( 1 / √ nM ). In comparison, the multi-head architecture from from D'Eramo et al. [26] requires additional encoder and decoder heads for every task, and thus, the cost of learning all the encoders and decoders is only O ( 1 / √ n), i.e. it is not reduced when increasing the number of tasks M , as it is in our case for URMA. In conclusion, as URMA only uses a single general encoder and decoder for all tasks, it compares favorably to the typical multihead approach as it can focus on learning only a single mapping to the shared representation space compared to the multi-head architecture which needs to learn M different encodings and decodings. This leads to the lower sample cost of learning these shared representations. 
experiments: 
 In this section, we evaluate our method from three different perspectives. First, we assess how well the model learns to control multiple embodiments in parallel against classical MTRL baselines. Then, we analyze how well the URMA architecture performs in terms of zero and few-shot transfer. Finally, we test the deployment capabilities of our learning framework and control architecture on real robots, allowing us to bridge the sim-to-real gap effectively. For all experiments in simulation, we will use the following two baselines.-Multi-Head Baseline. One way of dealing with the issue of different action and observation space sizes is to use a multi-head architecture with an encoder head for every environment, a shared core, and a decoder head for every environment [26] . We implement this baseline by using one shared encoder and decoder for all quadruped robots, one for all humanoid and bipedal robots, and one for the hexapod. The observations of all robots with identical morphology are arranged in the same order for the encoder, and the observations for missing joints are simply set to zero, e.g. humanoids often have different joint configurations. Compared to URMA, this baseline allows for efficient learning as the morphology-specific heads inherently separate the observations and arrange them in the correct order from the beginning. However, introducing new joints or completely new morphologies requires adding new neurons to a head or training a completely new head from scratch. Every additional head is a new mapping into and out of the shared representation space, which leads to a higher learning complexity compared to URMA.-Padding Baseline. Another way to handle differently sized action and observation spaces is to pad the observations and actions with zeros to fit a maximum length [28] . Therefore, a specific observation dimension can now represent different things for different robots. We add a one-hot task ID to the observations to ensure that the policy can distinguish between the robots, as typically done in MTRL [27] . Compared to URMA, the padding baseline is less complex but has similar issues when transferring to new morphologies like the multi-head baseline, as a new robot essentially represents a completely new task, and the policy has a hard time transferring knowledge between the differently structured observations and actions.To train our locomotion policy, we use the CPU-based MuJoCo physics simulator [43] for 16 different simulated robots with three learning environments each, resulting in 48 parallel environments in total. Figure 1 shows all the simulated robots we utilize for training and the three real robots that we use for deployment in the real world. The set of robots includes nine quadrupeds with three different joint configurations, five humanoids with five different joint configurations, one biped, and one hexapod. To leverage the huge amounts of data that we can generate in simulation, we use the PPO algorithm [44] . We build on the codebase of the DRL library RL-X [45] to implement our architecture and the baselines in JAX and to run the experiments. All hyperparameters used for URMA and the baselines are noted in Table 4 in Appendix B.First, we want to evaluate the training efficiency of MTRL in our setting. We train URMA and the baselines on all robots simultaneously and compare the average return to the single-robot training setting, where a separate policy is trained for every robot. All policies are trained on 100 million steps per robot, and every experiment shows the average return over 5 seeds and the corresponding 95% confidence interval. Additionally, we plot the empirical maximum performance when continuing the training for 1.6B steps on only a single robot as a dashed line. The performance on all 16 robots individually can be found in Figure 13 in Appendix L. Figure 3 confirms the advantage in learning efficiency of MTRL over single-task learning, as URMA and the multi-head baseline learn significantly faster than the average over training only on a single robot at a time. As expected, early on in training, URMA learns slightly slower than the multi-head baseline due to the time needed by the attention layers to learn to separate the robot-specific information, which the multi-head baseline inherently does from the beginning. However, URMA ultimately reaches a higher final performance. The padding baseline performs noticeably worse than the other two. We argue that the policy has trouble learning the strong separation in representation space between the different robots-which is necessary for the differently structured observation and action spaces-only based on the task ID.Next, we evaluate the zero-shot and few-shot transfer capabilities of URMA and the baselines on two robots that were withheld from the training set of the respective policies. We test the zero-shot transfer on the Unitree A1, a robot whose embodiment is similar to other quadrupeds in the training set. Figure 3 shows the evaluation for the A1 during a training process with the other 15 robots and highlights that both URMA and the multi-head baseline can transfer perfectly well to the A1 while never having seen it during training. The policy trained only on the A1 (shown in black) performs distinctly worse during the 100M training steps as it needs more samples per robot to learn the task. It eventually catches up to the multi-embodiment zero-shot performance while training for 1.6B steps directly on the A1. Further ablations on the minimal set of quadrupeds needed for good zero-shot capabilities on the A1 can be found in Appendix H. To investigate an out-of-distribution embodiment, we use the same setup as for the A1 and evaluate zero-shot on the MAB Robotics Silver Badger robot, which has an additional spine joint in the trunk and lacks feet observations, and then fine-tune the policies for 20 million steps only on the Silver Badger itself. Figure 3 shows that URMA can handle the additional joint and the missing feet observations better than the baselines and is the only method capable of achieving a good gait at the end of training. After starting the fine-tuning (gray vertical line), URMA maintains the lead in the average return due to the better initial zero-shot performance. Similar zero-shot and fine-tuning experiments for the humanoid morphology can be found in Appendix I.To further assess the adaptability of our approach, we evaluate the zero-shot performance in the setting where observations are dropped out, which can easily happen in real-world scenarios due to sensor failures. To test the additional robustness in this setting, we train the architectures on all robots with all observations and evaluate them on all robots while completely dropping the feet observations. Figure 3 confirms the results from the previous experiment with the Silver Badger and shows that URMA can handle missing observations better than the baselines.Finally, we deploy the same URMA policy on the real Unitree A1, MAB Honey Badger, and MAB Silver Badger quadruped robots. Figure 1 and the videos on the project page show the robots walking with the learned policy on pavement, grass, and plastic turf terrain with slight inclinations. Due to the extensive DR during training, the single URMA policy trained on 16 robots in simulation can be zero-shot transferred to the three real robots without any further fine-tuning. While the Unitree A1 and the MAB Silver Badger are in the training set, the network is not trained on the MAB Honey Badger. Despite the Honey Badger's gait not being as good as the other two robots, it can still locomote robustly on the terrain we tested, proving the generalization capabilities of our architecture and training scheme.Limitations. While our method is the first end-to-end approach for learning multi-embodiment locomotion, many open challenges remain. On one side, our generalization capabilities rely mostly on the availability of data, therefore zero-shot transfer to embodiments that are completely out of the training distribution is still problematic. This issue could be tackled by exploiting other techniques in the literature, such as data augmentation and unsupervised representation learning, to improve our method's generalization capabilities. Furthermore, we currently omit exteroceptive sensors from the observations, which can be crucial to learning policies that can navigate in complex environments and fully exploit the agile locomotion capabilities of legged robots. Lastly, as humanoid robots only recently started to be available for reasonable prices, we could not test the deployment on one of these platforms yet. Better modelling, more reward engineering or additional randomization might be necessary to ensure their real-world transfer. 
conclusion: 
 We presented URMA, a open-source framework1 to learn robust locomotion for different types of robot morphologies end-to-end with a single neural network architecture. Our flexible learning framework and the efficient encoders and decoders allow URMA to learn a single control policy for 16 different embodiments from three different legged robot morphologies. We highlight URMAs learning efficiency in a theoretical analysis of its task-averaged risk bound and compare it to prior work. In practice, URMA reaches higher final performance on the training with all robots, shows higher robustness to observation dropout, and better zero-shot capabilities to new robots compared to MTRL baselines. Furthermore, we deploy the same policy zero-shot on two known and one unseen quadruped robot in the real world. We argue that this multi-embodiment learning setting can be easily extended to more complex scenarios and can serve as a basis for locomotion foundation models that can act on the lowest level of robot control. Finally, the URMA architecture is general enough to be applied to not only any robot embodiment but also any control task, making task generalization, also for non-locomotion tasks, an interesting avenue for future research. 
 
 Appendix: Randomization is necessary to ensure that learned policies can be transferred to the real world [1, 13, 46] . Therefore, we randomize as many aspects as possible to learn a robust policy for all robots in the training set and also to ensure that the policy can generalize well to otherwise unseen parameters during zero-shot transfer to new robots. We randomize the robot attributes (trunk mass and inertia, center of mass displacement, feet sizes, joint torque limits, joint velocity limits, joint dampings, rotor inertias, joint stiffnesses, joint friction coefficients and joint control ranges.), the terrain (ground friction coefficient, gravity, contact stiffness and damping), the control (PD gains, action scaling factor, motor strength, joint offsets and actuator delays) and the initial state of the robot (linear and angular velocity, orientation, joint positions and velocities). A new set of these variables is sampled twice per episode on average. Also, to make the policy robust to noisy sensors and learn to act even when some observations are missing, we add observation noise at every time step and randomly drop observations, i.e. set them to zero. Finally, to make the policy robust to external disturbances, we randomly perturb the linear velocity of the robots.The hyperparameters used for training the policies in our experiments are noted in Table 4 . We use the same hyperparameters for all architectures to ensure a fair comparison. For the fine-tuning experiments, we reduce the learning rate by 1/3 to make the learning process more stable and avoid forgetting in the policy by initial overfitting to the new robot. All multi-embodiment learning experiments are conducted on AMD Ryzen 9 or AMD EPYC CPUs and NVIDIA RTX 3090 or NVIDIA A5000 GPUs. For a single training run with all 16 robots and the hyperparameters noted in Table 4 , we allocate 20 CPU cores, 32GB of RAM and 1 GPU. With this setup, training the policy for 100 million steps per robot (1.6 billion steps in total) takes around 2.5 days. The CPU-based MuJoCo simulation with the small amount of parallel environments (48) poses the bottleneck in our training process. Moving to a more powerful GPU-based simulator with thousands of parallel environments would significantly speed up the training.To evaluate the effectiveness of the LayerNorm layers in the URMA architecture, we perform an ablation study where we remove all LayerNorm layers from the architecture. Figure 4 shows that removing the LayerNorm layers leads to a significantly slower learning process and lower final performance.In the URMA architecture, we use two separate encoders for encoding the joint description vectors d j for the joint observation encoding and the universal action decoding. In the joint encoder, we use the f ϕ MLP and in the universal decoder, we use the g ω MLP. To ablate the necessity having two separate encoders, we replace g ω with f ϕ in the universal decoder. None of the gradients are stopped, so f ϕ is optimized for both the joint encoder and the universal decoder. We test the two cases where the representation used for the universal decoder is either taken after the softmax operation (g ω = full f ϕ ) or before the softmax, i.e. after the tanh operation (g ω = partial f ϕ ). Figure 5 shows that using the same joint description encoder in both places leads to a slightly worse performance when using the full f ϕ representation and the same performance when using the partial f ϕ representation. Therefore, having two separate encoders for the joint descriptions does not seem to be necessary. However, it might add more flexibility, allowing the policy to learn two sets of encodings when needed.URMA uses the robot mass m and dimensions L, W and H in the general observations o g and as part of the joint and feet descriptions d j and d f . To make sure that the policy does not overfit to specific values of these attributes, we use strong randomization of the mass of the torso and slight randomization of joint nominal positions, which leads to slight changes in the robot bounding boxes (i.e. robot dimensions). To evaluate the necessity of even having the mass and dimensions in the observations, we remove them from the observations and descriptions. Figure 6 shows that removing the mass and dimensions from the observations and descriptions leads to the same performance. Therefore, the policy does not require the robot's mass and dimensions to learn strong locomotion for the different robots.Getting the most fine-tuned gait for every robot requires tuning the reward coefficients for all robots individually. Mainly, the relative weighting between the two command tracking terms and all the penalty terms needs to be adjusted. More inherently unstable platforms like humanoids need lower penalties at the start of training, to avoid the policy getting stuck in local minima when trying to stabilize and comply with the penalties at the same time. Furthermore, specific joint properties, e.g. joint accelerations or joint torques, need to take on larger values for bigger and heavier robots and also the penalties based on their squared norms (Table 2 ) scale with the number of joints of the robot. Nethertheless, we only tune 7 out of the 14 reward coefficients for all robots (Table 3 ), where many of these coefficients are set to the same or similar values for robots of the same morphology. The precise tuning of reward coefficients is not necessary to learn robust policies with good locomotion gaits, rather it is an engineering tool to get fine-grained control over the policy's final behavior.However, it is possible to use the same set of reward coefficients. To demonstrate this, we perform an ablation study where we train an URMA policy with the same set of reward coefficients for all robots and the same reward curriculum length (URMA: Single reward set) and compare it to a policy with our differently tuned reward coefficients (URMA: Multi reward sets). We evaluate the single reward set policy on all robots using the tuned reward coefficients (URMA: Single reward set, eval multi reward sets) for a fair comparison. We chose the reward coefficients and the reward curriculum length for the single reward set policy to be conservative (Table 5 ), i.e., we use high tracking coefficients and low penalty coefficients, to ensure that every robot will learn to walk. Robot T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 T11 T12 T13 T14 T All robots 5.0 2.5 2.0 0.05 0.2 0.2 10.0 2.5e-7 2e-5 6e-3 30.0 1.0 0.1 0.5 80e6 Table 5 : Reward coefficients r c for all the reward terms for all robots. The last column T is the time-dependent reward curriculum length until all penalties are linearly increased from zero to their final values. The description of the reward terms can be found in Table 2 .Figure 7 shows that using the same set of reward coefficients for all robots keeps the stable learning process and its final performance on the differently tuned reward coefficients is only slightly worse than the policy trained directly with the tuned reward coefficients. Therefore, using the same set of reward coefficients for all robots is a valid option to simplify the need for engineering going into the training process and still achieve good locomotion gaits.To evaluate the impact of the number of robots and specifically quadrupeds in the training set required for zero-shot transfer to a new quadruped, we perform an ablation study to find a minimal set of quadrupeds that are needed to achieve good zero-shot transfer performance on the Unitree A1. As the Unitree A1 is most similar to the Unitree Go1 and Go2 robots, we train a policy only on those two robots and test if they are already enough to transfer to the A1. Further, we add the Barkour v0 (Bkv0), Barkour vB (Bkvb) and the MAB Robotics Silver Badger (SBg) to the training set to see if the performance improves with these additional robots. Then, we train on all 8 quadrupeds besides the A1, i.e. including the ANYmal B and C and the Petoi Bittle. Finally, we train on the full set of robots besides the A1, so including also the biped, humanoids and hexapod robots. Figure 8 shows that training on the Unitree Go1 and Go2 robots already leads to some decent zeroshot transfer performance on the A1. Including the Barkour v0, Barkour vB and the MAB Silver Badger improves the performance further. Training on all quadrupeds besides the A1 leads to the best zero-shot transfer performance and is on par with training on all robots. Therefore, having strongly similar robots in the training set (Go1 and Go2) is enough to get some initial zero-shot transfer performance, but using the most diverse set of quadrupeds leads to the best transferability in the end.We evaluate the zero-shot and fine-tuning performance of URMA on the Unitree H1 and the PAL Robotics Talos humanoid robots by following the same procedure as for the MAB Robotics Silver Badger quadruped (Figure 3 bottom left). We first train the policy on all robots besides the H1 / Talos, while evaluating the zero-shot performance on the H1 / Talos, and then fine-tune the final policy on the H1 / Talos and evaluate the performance again. A standard URMA policy trained on all robots and one trained on only the specific robot from scratch are used as a baseline for comparison. Generalization across our set of biped and humanoid robots is challenging due to the small set of highly variable embodiments and the inherent instability of the humanoid form factor. Figure 9 and Figure 10 show this difficulty, as the policy is not able to do any initial transfer to the Unitree H1 or the Talos. We hypothesize that more biped and humanoid robots in the training set or additional robot randomization on the current robots are necessary for the policy to get some meaningful zeroshot transfer performance. Nethertheless, URMA learns strong general locomotion features during the initial training on other robots, including other humanoids, and is able to quickly adapt to the H1 / Talos during fine-tuning. The fine-tuned policy is able to learn quicker and achieve a higher performance in the 20M steps of fine-tuning than the multi-embodiment and single-robot baseline policies trained from scratch in the same short amount of time. While the fine-tuning policy and the policies trained from scratch will reach the same performance in the end, fine-tuning with an already good representation space allows for quick adaption and fast learning on a new embodiment.Having different robot morphologies in the training set can potentially hurt or help the performance on robots of a specific morphology, leading to inter-morphology generalization. We evaluate the impact of training on additional morphologies by comparing the performance for either quadruped or biped and humanoid robots of a policy trained on all robots with a policy trained only on the respective morphology.Figure 11 shows that training on additional morphologies does not hurt the performance in the case of quadruped robots and marginally decreases the performance in the case of biped and humanoid robots. We test the performance on robots that are already in the training set, so it is reasonable that the policy can perform best when training only on these robots. In the quadruped case, the policy seems to have enough capacity to also fully fit the humanoids and hexapod without losing performance. We think that inter-morphology generalization and reasoning about multiple morphologies can still occur and help the policy in the zero-shot and few-shot case, especially when encountering embodiments that mix properties of previously seen robots from different morphologies.The joint description vectors d j are an integral part in URMA. In the joint encoder, they are used to route the corresponding joint observations to the correct dimension in the joint latent vector zjoints .In the universal decoder, the joint descriptions are used to unpack the joint specific actions from the action latent vector zaction . To investigate how strongly the policy conditions on the joint descriptions, we first completely shuffle the joint descriptions for an already trained policy and test it on the Unitree A1 quadruped and the Unitree H1 humanoid. The A1 is able to stand for an all zero velocity command but when receiving any non-zero velocity command, it struggles and falls after one or two steps. The H1 is not able to stand at all and falls mostly immediately, due to the stronger inherent instability of the humanoid morphology. The behavior in both cases does not appear random but rather systematic and strongly uncoordinated between the joints. This makes sense as the policy learned smooth actions but the correct mapping is mixed up.We also test the conditioning of the policy on different attributes of the joint descriptions. We take the Unitree A1 and strongly perturb the joint descriptions (multiplying the associated values by either 1/3 or 3) for the PD gains + action scaling factor, the joints rotor inertias, and the joints friction coefficients. When we increase the values of the description of the PD gains + action scaling factor, the policy assumes the robot's motors and controller are stronger and more responsive than they are. The same situation occurs for the rotor inertia and friction coefficient, the policy assumes that the joints are less responsive and smooth, causing performance degradation. The left side of Figure 12 shows that decreasing the description values for the PD gains and action scaling factor by a factor of 1/3 leads to the policy trying to overcompensate compared to the normal values (factor 1) by producing actions with a higher average magnitude measured over 10 episodes. In the opposite case, increasing the values by a factor of 3 leads to the policy producing actions with a lower average magnitude, as it thinks the robot is more responsive than it actually is. A similar but less pronounced effect can be seen in the middle and right side of Figure 12 for the rotor inertias and friction coefficients. Lower perceived inertias and friction coefficients in the joints (multiplying the values for every joint with 1/3) leads to the policy producing actions with a lower average magnitude and vice versa. In general, the policy seems to be able to capture the changes in the different attributes of the joint descriptions and adapt its actions to their physical implications.L Individual Robot Returns This section provides a detailed proof of Theorem 1.First, we show that the loss function ℓ in ( 4) is normalized and bounded to [0, 1]. The PPO lossoptimized by gradient descent-for a single pair X mi , Y mi is defined as follows:EQUATIONwhere s, d and a are the observations, descriptions and actions coming from the sample X mi , π(a|s) = w(h(f (X mi ))), π 0 (a|s, d) is the fixed probability density of the first PPO iteration from Y mi and A π is the true advantage under π from Y mi . To bound L we need to bound A π and the ratio r = π(a|s,d) π0(a|s,d) in (9) . By looking at the definition of the advantage A π = Q π -V π , we can bound it between A min = Rmin 1-γ -Rmax 1-γ and A max = Rmax 1-γ -Rmin 1-γ , where R min and R max are the minimum and maximum possible reward. In our training environments, R min = 0 and R max is the reward given when the policy tracks the given target commands c perfectly. Therefore, in our setting A min = -Rmax 1-γ and A max = Rmax 1-γ . To bound r in (9), we need to look at the following four cases:• If A π > 0 and r < 1 then r is bounded by 1 -ϵ.• If A π > 0 and r > 1 then r is bounded by 1 + ϵ.• If A π < 0 and r < 1 then r is bounded by 1 -ϵ.• If A π < 0 and r > 1 then r is not directly bounded, but we can modify the PPO training to discard (s, d, a) pairs with too big probability ratios, i.e. only allowing samples with r < 1 + E where E is some upper probability ratio that we allow in the case of negative A π .With the bounds on A π and r, we can normalize and bound the loss function to be in [0, 1] by defining ℓ asEQUATIONwhere l min = -A max (1 + ϵ) = -Rmax 1-γ (1 + ϵ) and l max = -A min (1 + E) = Rmax 1-γ (1 + E). Because our encoder f and decoder w functions take sequences of joint and feet observations and descriptions as input, we rely on the attention mechanism in the encoder and our universal morphology decoder to flatten their respective outputs for the following functions. In practice, in the encoder, we sum over z j to get rid of the sequence dimension (see (2) ), and at the end of the decoder, we concatenate the resulting actions for every joint of a to get the flat vector a = [a 1 , a 2 , . . . , a J ].To ensure every vector is of the same length, we pad them with zeros.We start the proof of Theorem 1 by bounding the excess risk, i.e. the difference between the risk ε avg ( f , ĥ, ŵ) where f , ĥ, ŵ are the minimizes of the empirical risk, and the minimum task averagedrisk ε * avg . Notice that excess risk represents how much we pay in terms of performance w.r.t. the optimal policy update since we are optimizing our policy using only a finite amount of samples. We decompose the excess risk, as done in D'Eramo et al. [26] , as follows: ε avg ( f , ĥ, ŵ) -ε * avg = ε avg ( f , ĥ, ŵ) - To bound the complete difference, we will bound its three components A, B and C. First, B is bounded by 0, as f , ĥ and ŵ are the true minimizers of the empirical risk and therefore f * , h * and w * can not achieve a lower loss on the empirical dataset. Next, C can be bounded with Hoeffding's inequality with probability 1 -δ 2 by ln( 2 /δ) (2nM ) , as it contains only nM random variables bounded in the interval [0, 1]. Finally, to bound term A, we begin with defining the following auxiliary sets, which make the rest of the derivations cleaner:• S = L(W(H(F( X))), Ȳ) = {(ℓ(w(h(f (X mi ))), Y mi )) : ℓ ∈ L, w ∈ W, h ∈ H, f ∈ F} ⊆ R M n • S ′ = W(H(F( X))) = {(w(h(f (X mi )), Y mi )) : w ∈ W, h ∈ H, f ∈ F} ⊆ R M n • S ′′ = H(F( X)) = {(h(f (X mi )), Y mi ) : h ∈ H, f ∈ F} ⊆ R M n × XWe start to bound A with probability 1 -δ using Theorem 9 from Maurer et al. [41] :ε avg ( f , ĥ, ŵ) -1 nM With the contraction lemma, Corollary 11 of Maurer et al. [41] , we are bounding G(S) ≤ L(ℓ)G(S ′ ). Compared to Maurer et al. [41] , we can't guarantee that the PPO loss is 1-Lipschitz, and therefore, we keep the Lipschitz constant of L(ℓ). In practice, this is no problem, as it is just a constant that can be reduced by scaling the loss. Furthermore, we bound the resulting Gaussian complexity of S ′ with Theorem 12 from Maurer et al. [41] , given the universal constants c ′ where L(W) is the largest value for the Lipschitz constants in the decoder function space W, and D(S ′′ ) is the Euclidean diameter of the set S ′′ . We finish the bounding of the Gaussian complexities of the auxiliary sets by using Theorem 12 from Maurer et al. [41] once more. For the universal constants c ′′ Project page: https://nico-bohlinger.github.io/one_policy_to_run_them_all_website/ 


Decision: **Post-rebuttal metareview**:

This paper proposes a novel locomotion policy architecture, URMA (Unified Robot Morphology Architecture), which learns a single policy for different types of legged robots. The cross-embodiment capability and the proposed architecture are pretty novel. Most of the reviewers' concerns are addressed in the rebuttal. Adding more real-world results for non-quadrupedal robots will further strengthen the paper.

----------------
**Pre-rebuttal metareview**:

Strengths:
1. The proposed idea to solve cross-embodiment locomotion is novel and interesting.
2. Real-world experiments on quadrupeds are promising.

Weakness:
1. Presentation needs improvement.
2. Need more fair and comprehensive comparison. Why is the specialized policy worse than the main policy?
3. No real-world experiments for non-quadruped robots, which is the main point of the paper.
4. The “first step towards foundation model” argument needs justification.
5. Other issues mentioned by reviewers such as presentation, training setups, lack of ablation, etc.

Reviews: review: title: title: Learning Multi-Embodiment Locomotion based on Multi-Task Reinforcement Learning
paper_summary: paper_summary: The paper proposes the Unified Robot Morphology Architecture (URMA) to learn a single locomotion policy for robots with different morphologies. This study uses the Multi-Task Reinforcement Learning (MTRL) concept to share knowledge between tasks and learn a common representation space applicable to all tasks. In the proposed method, embodiment-specific observations are encoded and combined into a single joint latent vector to handle the varying sizes of observations for each robot. To determine the final action for the robot, a universal morphology decoder is employed. This decoder takes the general action latent vector, pairs it with the set of encoded specific joint descriptions, and combines it with the single joint latent vectors to produce the mean and standard deviation of the actions for every joint. Overall, the proposed idea is intriguing, and targeting the challenge of cross-embodiment learning is a nice effort. However, the paper's presentation, comparison, and quantification need significant improvement.
main_review: main_review: The paper proposes the Unified Robot Morphology Architecture (URMA) to learn a single locomotion policy for robots with different morphologies. This study uses the Multi-Task Reinforcement Learning (MTRL) concept to share knowledge between tasks and learn a common representation space applicable to all tasks. In the proposed method, embodiment-specific observations are encoded and combined into a single joint latent vector to handle the varying sizes of observations for each robot. To determine the final action for the robot, a universal morphology decoder is employed. This decoder takes the general action latent vector, pairs it with the set of encoded specific joint descriptions, and combines it with the single joint latent vectors to produce the mean and standard deviation of the actions for every joint. Overall, the proposed idea is intriguing, and targeting the challenge of cross-embodiment learning is a nice effort. However, the paper's presentation, comparison, and quantification need significant improvement.

***Comments and Questions***

1.  The idea of using Multi-Task Reinforcement Learning for cross-embodiment is nice, as the framework can handle varying sizes of observation and action spaces. The experiment with quadrupeds looks promising.

2.  It is counter-intuitive that shown, in the paper, the specialized policy training performs distinctly worse than the main policy. It is well-established in the literature that specialized policies can perform agile and challenging locomotion tasks effectively. This limitation should be addressed, as maybe the proposed framework is not suitable for specialization. For a fair comparison, the proposed method should be compared with a simple policy using MLP and the current state-of-the-art RL locomotion techniques (that obviously shows impressive performance in the literature).

3. Comprehensive comparison and quantification of learned gaits in terms of energy efficiency, robustness, and agility should be conducted with a single trained policy and a specialized policy for a single robot with state of the art RL schemes.

4. Surprisingly, there is no video of other embodiments' locomotion except quadruped, which is a central point of the paper. This is a critical issue for conference submission, as all submitted papers should be reviewed under fair and equal conditions, and videos should not be submitted after rebuttal. How can readers understand the locomotion behavior of the other embodiments solely from the reward plot?

5. The authors state this is the first step toward a foundation model for locomotion. "This flexible architecture can be seen as a first step in building a foundation model for legged robot locomotion." I believe this is not the first step in building a foundation model, as there are many other recent efforts in this area. For example please see the following reference as a step toward cross-embodiment of locomotion:

- `Luo, Zeren, et al. "MorAL: Learning Morphologically Adaptive Locomotion Controller for Quadrupedal Robots on Challenging Terrains." IEEE Robotics and Automation Letters (2024).`

6. Regarding the total mass of the robot \( m \), its length \( L \), width \( W \), and height \( H \) being part of the observation space, please mention in the limitations that this information is required for each embodiment. This implies that the network implicitly receives a token for each robot. What about adding a robot with different mass and dimensions that the network has not experienced during training? These parts of the observation space would be out of the training distribution and could lead to unexpected actions.

7. What is the difference between the observation of height above ground \( h \) and height \( H \)? Please clarify this point.

8. What are the smallest and largest robots that can be controlled during locomotion in terms of mass and height? Please provide specific measurements.

9. One limitation of the work is that it requires reward weight tuning for each embodiment. Please mention this in the limitations section.

10. How is the computational complexity? How many hours does it take to train the policy for these 16 robots?

11.  The architecture figure is visually appealing but somewhat confusing. Please improve their clarity and fluency.

12.  Since the paper aims to take a step toward a foundation model for robotics, it is suggested to open-source the code to join the forces with different research groups.
questions: questions: Please review the questions mentioned in the review section and check the comments.
limitations: limitations: 1
review_summary: review_summary: After considering the rebuttal, I have updated my decision to "weak accept."
title: title: Strong step towards general control policies for arbitrary robot configurations.
paper_summary: paper_summary: This paper presents a method for learning a general locomotion policy that can be used on multiple diverse robot platforms, varying from quadrupeds to humanoids to hexapods. This abstract controller can then be zero-shot transferred to unseen platforms of similar nature to those seen in training, i.e. a different unseen quadruped robot. By encoding the information of each robot joint individually and then combining them with an attention head into a latent vector, networks of the same size can be used regardless of the number of joints each robot has. The authors then train their system with 16 different robots and show that it can zero-shot transfer to unseen platforms that are similar nature and can be quickly fine-tuned if the morphology differs a lot.
main_review: main_review: Overall I think the work is strong and is moving towards an exciting and valuable direction. While I am not fully convinced yet that the results shown here can fully generalize to a diverse set of robots and tasks, the method presented is arbitrarily scalable and definitely has the potential to be general, something that prior works do not have. The results do still seem promising and provide a direction to work towards in having a large robot model that others can use to fine tune for their own platform. This is something that can benefit many researchers in speeding up training time for their own works.

## Strengths:

The proposed method has a truly general way of encoding both the joint (state) space and action space of arbitrary robot morphologies/configurations. The method does not rely on any padding or per task/robot networks and thus has the potential to scale arbitrarily, something that I have not seen from previous works. This is the main strength over previous MTRL works. Given more robots to train over, the number of network parameters does not change, allowing for an actual useable framework for a robot “big data” regime where having more data should only help performance. 

I can see the impact of this work being big. If there are more refinements, and it can allow for offline RL use, it can allow for lots of sharing and reuse of good performing policies across the field. Individuals can contribute their own robot datasets, which can be used to improve a large robot foundation model, and can then use it and fine-tune for their own specific platform.

The text and method are for the most part clear and understandable. Explanation details are mostly thorough and go into the sufficient greater detail in the appendix.

The number of training experiments run is appreciated, training policies for 16 works is a lot of work. Though I would like to see some additional comparison experiments (see below).

The technical details of the methods and analysis of the algorithm seem to be all correct to my understanding.


## Limitations:

I think the explanation of what the “joint description vector” is exactly should be moved from the Appendix into the main text. These design choices to me seem highly important, since this is basically your choice of robot agnostic abstraction. How you are choosing to describe robots in a generic way is a main part of your contribution. There is also no exact explanation of what the “feet description” vector contains. That is definitely needed as well.

It would also be interesting to see how much performance are you giving up in favor of generality, if any at all, or even in the extreme case the policy seeing different morphologies is helpful. Presumably, if instead of spending 100 million steps on each robot (so 1.6 billion steps total) you spent all 1.6 billion steps specializing for just a single robot you would get higher performance. How much higher performance is this and is the performance actually higher? Or even if given the same data budget is seeing different morphologies more helpful?

It would also be interesting and useful to see some study on how many robots are needed to get good generality. From your results it seems like you can train on 8 different quadrupeds and still zero-shot generalize to a similar quadruped. Can this still happen if you train on less quadrupeds? I guess the zero-shot test to the Silver Badger kind of shows this; in that case since the Silver Badger is a more different kind of quadruped it’s zero-shot performance is worse. But I would be interested in some experiment to see if there is a “minimal set” of quadrupeds you can train over while still having satisfactory generalization performance.

In a similar line, how much does vastly different morphologies (like quadruped vs. biped/humanoid) help learning of other morphologies? If you remove bipeds and humanoids from the training distribution how does that affect the results on the quadrupeds? So if I only care about generalization to/fine-tuning for a more specific class of robots like quadrupeds or humanoids should I still care about training on different classes of robots?

There are also no generalization experiments done for humanoid robots. If you do similar zero-shot and fine-tuning experiments on one of the humanoid robots as you did for the A1 and Silver Badger what would the performance look like? I think this is necessary to have, in order to show the generalization and quick fine tuning that you argue your method has, and is the entire point of your method in the first place.

The current work is also limited to just diverse robot morphologies, not different tasks (which is understandable, asking for robot and task generalization is a lot for a single work). I would be interested to see if this same method of “foundation model” can provide speed up when fine-tuning for different non-locomotion tasks.

Overall I think the main limitation is that not enough generalization and fine tuning experiments are shown to convince me of the main purpose of this method: that it allows quick fine tuning/generalization to new diverse robots. I would like to see more of the evaluation experiments done for other robot morphologies besides quadrupeds.
questions: questions: Is there no MLP encoding for the feet observations/descriptions? (There is no MLP blocks there in Figure 2, they go straight into the attention encoding)

Why is there a different joint description encoding used to produce the mean and std dev of the action? Why can’t the same $f_\phi$ encoding be used?

Why do the single-robot training baselines do so poorly on the quadruped scenarios? You mention that this is because of aggressive DR in order to facilitate sim2real transfer, but from the learning curves it looks like they won’t learn at all, not just that they require more samples (the learning curve seems to have plateaued already). Is this the case or if given more samples will the single-robot baselines actually learn? If not then that is very strange since there are many multiple prior works (that you reference, [1-10]) that can solve the quadruped locomotion task.

Can you provide ranges for the DR used, especially since there seems to be some odd choices of things to randomize? Why would feet size and torque and velocity limits need to be randomized, these should all be pretty well known and fixed right? What is the “action scaling factor”, is this literally some scaling of the policy action? What is the point of that, even if the resulting torque is different on hardware, we know absolutely for certain what action we are trying to command right? What is “motor strength”, is this like some torque scaling factor? Would the randomization of the PD gains not have the same effect as this?
limitations: limitations: 3
review_summary: review_summary: A good step towards the possibility of an actual "robot foundation model". Novel control architecture that can scale and handle robots with arbitrary number of joints/actuators. Shows successful zero-shot transfer and fine-tuning capabilities in limited settings with possibility for more generalization.
title: title: review
paper_summary: paper_summary: This paper presents a archetecture to train a unified policy that can control many different legged robots.
main_review: main_review: This paper demonstrates a system that can learn a unified policy that can control legged robots across different morphologies and sizes. 

Strength
Demonstrate that the policy can either transfer zero-shot to a similar robot or can be fine-tuned more efficiently on out-of-distribution robots compared to baseline. The policy also seems to work pretty well on some real quadruped robots.

Weakness
Overall, it seems like a complicated system. How much tuning is needed in the reward function for a new robot to work is unclear. It seems like different robot needs different reward coefficients, so the benefit of a unified policy is kind of limited. Since Go1 and A1 are pretty similar, it is not surprising that the policy can transfer to A1, it would be interesting to investigate the use of joint description. If one changes the description by a lot, will the policy actually make use of that information to make dramatic changes to the output?

There is also a lack of ablations on how many robots are needed to train the initial policy to ensure a good zero-shot capability or as a good basis for finetuning.
questions: questions: (1) I don't understand the theoretical analysis part of the paper. What property of the URMA is used in the analysis that other baseline methods do not have? It would be great to explicitly state things like: "Our URMA has this property XXX, and using the property, we can derive this bound (without this property the bound would not work). While other architectures lack this property."

(2) The single robot training baseline performs very badly. Most papers in RL for quadruped use single robot training and they can demonstrate pretty good performance. I wonder where is this gap coming from.

(3) What do the motions of the bipedal robots look like? How important is it to include them for training if all I care about is quadruped robots? How about fine-tuning performance on bipedal robots?

(4) For the quadruped robot with a spine, is the policy fine-tuned to utilize the spine in any way? e.g., does it discover a strategy that quadruped with no spine cannot do?
limitations: limitations: 3
review_summary: review_summary: I think more analysis/evidence for the benefit of using a unified policy is needed.