MeMo: Meaningful, Modular Controllers via Noise Injection

Authors: megan  tjandrasuwita
jie  xu
wojciech  matusik
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-09 20:34:39

Abstract: Robots are often built from standardized assemblies, (e.g. arms, legs, or fingers), but each robot must be trained from scratch to control all the actuators of all the parts together. In this paper we demonstrate a new approach that takes a single robot and its controller as input and produces a set of modular controllers for each of these assemblies such that when a new robot is built from the same parts, its control can be quickly learned by reusing the modular controllers. We achieve this with a framework called MeMo which learns (Me)aningful, (Mo)dular controllers. Specifically, we propose a novel modularity objective to learn an appropriate division of labor among the modules. We demonstrate that this objective can be optimized simultaneously with standard behavior cloning loss via noise injection. We benchmark our framework in locomotion and grasping environments on simple to complex robot morphology transfer. We also show that the modules help in task transfer. On both structure and task transfer, MeMo achieves improved training efficiency to graph neural network and Transformer baselines.

Main paper: 
 introduction: 
 Consider the following scenario: A roboticist is designing a robot with 6 legs, such as the one seen in the left image of Fig. 1 , and has trained a standard neural network controller with deep reinforcement learning (RL) to control the actuators circled in green. However, after more testing, they realize that the design of the robot needs to be extended with another pair of legs to support the desired amount of weight. Even though the new 8 leg robot is still composed of the same standard assemblies, the roboticist is unable to reuse any part of the 6 leg robot's controller. While many works [1, 2, 3] have studied structure transfer, or transferring neural network controllers to different robot morphologies, these works take a purely data-driven approach of training a universal controller on a dataset representative of the diversity and complexity of robots seen in testing. In contrast, we desire to learn transferable controllers from only a single robot and environment, obviating the requirement for a substantial training dataset and resources to perform multi-task RL. Our experiments demonstrate that state-of-the-art approaches for transferring control to environments with incompatible state-action spaces struggle to generalize in this highly data-scarce setting. Motivated by the above scenario, we propose a framework, MeMo, for pretraining (Me)aningful (Mo)dular controllers that enable transfer from a single robot to variants with different dimensionalities. Learning transferable modules from a single robot trained on a single task is challenging, even when we focus on transfer among robots with similar global morphologies. The key insight MeMo leverages is that a robot is built from assemblies of individual components, such as the leg of a walking robot or the arm of a claw robot. These assemblies are specified by a domain expert who is able to account for the constraints imposed by the robot's hardware implementation in their specification. Given this information, MeMo learns assembly-specific controllers, or modules, responsible for coordinating the individual actuators that comprise a given assembly, which are coordinated by a higher-level boss controller. As we are able to reuse the modules when transferring to a robot built from the same assemblies, the problem of learning a controller for a different morphology boils down to learning the coordination mechanics among assemblies, rather than having to coordinate at the granular level of individual joints. Returning to the 6 leg robot in Fig. 1 , we see that the robot is comprised of multiple "leg" and "body" assemblies, circled in red and blue respectively in the right image. Module parameters are shared between assemblies of the same type, providing multiple training instances that help our modules generalize. After training the modules with MeMo, the "leg" and the "body" modules can then be reused to speed up the training of a different robot's controller, such as an 8 leg robot.To achieve this improved training efficiency, a key challenge is to balance the labor between the boss controller and the modules. In one direction, to prevent the modules from becoming too robotspecialized, we introduce information asymmetry into our architecture, where the modules are limited to seeing the local observations of the actuators that belong in the module. In the other direction, controlling the assembly through the module must be simpler than controlling the assembly directly, since otherwise there is no benefit to this new architecture. This is achieved by a new modularity objective (Section 2) that forces the modules to capture as much of the coordination mechanics within a subassembly as possible, given limited local observations. In practice, we use noise injection (Section 3.1) to optimize the new objective simultaneously with standard behavior cloning loss.To evaluate the transferability of the learned module, we apply MeMo in locomotion and grasping domains. We design two types of transfer: generalizing to more complex robot structures and to different tasks. When transferring model weights from a simpler agent, we show that MeMo significantly improves the sample efficiency of performing RL on more complex agents. We compare our framework with NerveNet, an alternative approach for one-shot structure transfer [4, 5] , and MetaMorph [3] , an approach for learning universal controllers. Our experiments show that MeMo either exceeds or matches NerveNet and MetaMorph's training efficiency during transfer, as the message-passing policies are prone to overfitting during pretraining. 
motivation for modularity objectives: 
 Our goal is to maximize the extent to which the assembly-specific modules take responsibility for the behavior of the robot. In this section, we formalize the objectives that our training pipeline should achieve. Given an expert controller F, one can train a modular controller, consisting of a higher-level B(oss) module that sends a signal to each of the W(orker) modules, to mimic F's behavior using the standard behavior cloning objective. Let B be parameterized by θ and W be parameterized by ϕ. For simplicity, this section assumes that there is a single W module. Definition 2.1. Behavior Cloning Objective. For a system with states s i ∈ S, the modular policy whose output is W ϕ (B θ (s i )) imitates the expert monolithic policy F.argmin θ,ϕ E i (W ϕ (B θ (s i )) -F(s i )) 2 (1)The behavior cloning objective ensures that the composition of modules can perform the desired task, but it is not enough to ensure that the worker module is useful. In software engineering, a component is most useful if it can provide a narrow interface to a rich set of functionality. In the context of modularity, this is analogous to W giving B only a few degrees of freedom to control the system's outputs; otherwise, a module W that gives B full control over the actuators will leave W with no real responsibility for the robot's behavior. Then, when W is used with a new robot or for a slightly different task, B needs to relearn all the details of how to control the output for that new setting.For example, consider a robot arm with 5 degrees of freedom that controls a lever in Fig. 2 . An ideal worker module would take as input a signal corresponding to the desired angle of the lever and would be responsible for coordinating the signals to the five actuators to achieve the lever's desired position. This would mean that if we want to reuse W in controlling two arms, the new boss B' will only have to learn how to coordinate the two angles, and not all 10 actuators. In practice, though, forcing the interface between B and W to be a one dimensional vector makes the optimization problem very difficult. Instead, our approach will be to use a larger vector as the interface between the two modules, but introduce an additional optimization objective. Intuitively, the interface is effectively narrow when B's signal can be decomposed into a small set of dimensions that result in greater variance in W's output and a much larger set of dimensions that do not cause significant perturbations in W's output. In other words, the null space of W's Jacobian would be higher-dimensional, meaning that W has a greater tolerance for error in B's signal that fall in the directions of the null space. To encourage W to be less sensitive to perturbations in B's signal, we minimize the distance between W ϕ (B θ (s i ) + η), where η is a noise vector, and W ϕ (B θ (s i )). Definition 2.2. Invariance to Noise Objective. Let η be a noise vector. The difference between the result of applying W on the distorted input and on the undistorted input isD(s i , η) = W ϕ (B θ (s i ) + η) -W ϕ (B θ (s i )).As a new distortion to B θ (s i ) is added on each epoch, we average the difference over the added noise.EQUATIONIn practice, we sample η from a Gaussian distribution with E[η] = 0 and E[η T η] = σ 2 I, where σ = 1.0. In Section 5.3, we demonstrate that our invariance to noise objective is the critical component in our framework that yields positive transfer benefits. In Section 5.4, we show that optimizing the noise invariance objective reduces the effective dimensionality of B's signal.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: See Appendix A.1. 
method: 
 We describe our approach MeMo, an algorithm for learning reusable control modules. In Section 3.1, we show that the modularity objectives can be optimized with noise injection. In Section 3.2, we extend our formulation to systems with more than one module and detail our training pipeline.We propose to optimize both modularity objectives simultaneously with noise injection. Definition 3.1. Noise Injection Objective. Here, η can be viewed as "injected noise."EQUATIONThe noise injection loss can be decomposed as follows:EQUATIONSee Appendix A.4 for the derivation of the decomposition. Without the last term, which we call the product term, noise injection is equivalent to the sum of Eq. 1 and 2. Analyzing the product term further, by the Mean Value Theorem,D(s i , η) = W ϕ (B θ (s i ) + η) -W ϕ (B θ (s i )) = J W (z) ⊺ η for z ∈ L,where J W denotes the Jacobian of W with respect to B's output and L is the line segment with B θ (s i ) and B θ (s i ) + η as endpoints. Applying the expectation over the noise:EQUATIONNote that z depends on the value of η, so it cannot be pulled out of the expectation. However, in practice, we expect that J W (z) ⊺ ≈ J W (B θ (s i )) ⊺ . This implies that E η [D(s i , η)] ≈ J W (B θ (s i )) ⊺ E η [η] = 0, making the product term negligible. Empirically, in Fig. 3 , we show that this product term indeed becomes much smaller than the sum of the two modularity objectives as training proceeds. For all starting morphologies, the modularity objectives dominate the loss as the ratio is less than 1 for all updates.Modular Architecture. Although thus far we have only a single module W, a robot is often comprised of multiple modules controlling physical assemblies that are common among different morphologies. Formally, we assume that we are given a partitioning P of an agent's joints j 0,...,N -1 .We design a modular policy composed of a boss controller B that outputs intermediate signals to neural network modules that decode actions. Each element of the partition, e.g. a subset of actuators, is a module instance i of type k, which we denote as W i k . In total, there are |P| modules. Modules of the same type k share the module parameters, yet each instance will receive a different message from B. We detail our architecture further in Appendix A.3.Training Pipeline. To train our modules, inspired by previous works that combine RL and IL [6, 7, 8] , we first train F using RL. During the RL stage, we use proximal-policy optimization [9] to train actor-critic controllers. The critic is a MLP, whereas the actor is a standard MLP when training the expert controller and a modular architecture when transferring pretrained modules. Once F is trained, we train a modular policy π θ,ϕ (a i | s i ) with IL via DAgger [10] , with noise injected into B θ 's output. At each iteration k of DAgger, we sample a trajectory D k from π θ,ϕ . F provides the correct action to each s ∈ D k , and D k is aggregated into the full dataset D = {(s i , a i )}. To optimize the objective defined in Section 3.1, we minimize L = -E si∼D [log π θ,ϕ (a i | s i )]. We derive a decomposition of the negative log likelihood loss with noise injection into the modularity objectives in Appendix A.5. After transferring the modules to a new structure or task, we perform RL to retrain B or finetune the architecture end-to-end. Our pipeline is summarized in Fig. 4 . Appendix A.6 details our RL and IL hyperparameter settings. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
related work: 
 Our work relates to modular controllers and structure transfer. Related works in noise injection, multi-robot RL, and hierarchical RL are discussed further in Appendix A.11.Modular Controllers. Our work relates to prior works that train modular policies for robot designs. [11] learns neural network policies that are decomposed into "task-specific" and "robot-specific" modules and performs zero-shot transfer to unseen task and robot-specific module combinations. [1] coordinates modular policies shared among all actuators via message passing. [12] uses a GNN to internally coordinate between part-specific nodes with shared module parameters between nodes corresponding to the same part. [13] proposes the Dynamic Graph Network to control self-assembling agents, consisting of modules that are shared across agents.Structure Transfer. In the hierarchical RL setting, [14] uses imitation learning to train policies that represent long-horizon behavior and improve sample efficiency when transferred from simple to complex agents. [15] transfers policies to robots with significantly different kinematics and morphology by defining a continuous evolution from the source to the target robot. Previous works use message-passing policy architectures to generalize across morphologies [1, 4, 5] . In the multitask setting [2] proposes Transformers as policy representations that remove the need for multi-hop communication. [3] scales Transformer-based policies to larger and more diverse datasets of robots. 
experiments: 
 With our experiments, we seek to answer four questions. 1) Do the modules produced by MeMo generalize when transferred to different robot morphologies and tasks? 2) When pretraining modular controllers with imitation learning, does the Gaussian noise injection help? 3) In the pretraining phase, why do we use imitation learning rather than injecting noise in reinforcement learning? 4) How does our modularity objective yield better representations of the actuator space? We answer question 1) in Sections 5.1 and 5.2, 2) and 3) in Section 5.3, and 4) in Section 5.4.We benchmark our framework on two types of transfer: structure and task transfer. While our framework is designed primarily for structure transfer, we use task transfer experiments as an additional means of evaluating the quality of the learned representations. For the locomotion experiments, we perform experiments on the tasks introduced in RoboGrammar [16] with training statistics computed as the average reward across 3 runs, with standard deviations indicated by shaded areas. For the grasping domain, we construct object-grasping tasks using the DiffRedMax simulator [17] and compute training statistics as the average reward across 5 runs. Additional details on the reward functions used are in Appendix A.9. We visualize train and test robot morphologies for structure transfer in Fig. 5 and the train and test tasks for task transfer in Fig. 18 .Locomotion. We design three structure transfer tasks in the locomotion domain, in which the goal is to move as far as possible while maintaining the robot's initial orientation. The starting morphologies are the 6 leg centipede robot, the 6 leg worm robot, and the 6 leg hybrid. The 6 to 12 leg centipede transfer demonstrates scalability to transfer robots with many more modules than seen in training. The 6 to 10 leg worm shows that MeMo generalizes with only 1-2 instances of the same module seen in training. The 6 and 10 leg hybrid robots involve three types of modules, demonstrating scalability to more complex training robots. For task transfer, we transfer policy weights pretrained on a 6 leg centipede locomoting over the Frozen Terrain to three terrains that feature obstacles or climbing.Grasping. In grasping, the goal is to grasp and lift an object as high as possible. We design a grasping robot consisting of an arm that lifts a claw grasping a cube. The structure transfer is from a 4 finger to a 5 finger claw. For task transfer, we transfer policies trained to control the 4 finger claw grasping a cube to the same robot grasping a sphere of similar size and weight.Figure 5 : Structure Transfer Tasks. Left: Transfer "leg" and "body" modules from a 6 to a 12 leg centipede. Left Middle: Transfer "body" and "head" modules from a 6 to a 10 leg worm. Right Middle: Transfer "leg," "head," and "body" modules from a 6 to a 10 leg hybrid. Right: Transfer "arm" and "finger" modules from a 4 to a 5 finger claw.Baselines. We compare MeMo to MLP and modular policies trained from scratch as well as pretrained NerveNet [4, 5] and MetaMorph [3] baselines. NerveNet takes as input the underlying graph structure of the agent, where the nodes are actuators and edges are body parts. The graph structures of the train morphologies are detailed in Appendix A.13. For MetaMorph, a Transformer-based approach, we convert the global observations and local observations for each actuator to a 1D sequence of tokens. Full training details and state space descriptions are included in Appendix A.6 and A.7 respectively.• RL (MLP): For structure transfer, due to the change in the observation space, we train a 2 layer MLP policy from scratch with RL. In task transfer, we use a MLP pretrained with RL on the original task and finetune it on the test task. For a fair comparison, we use the same architecture size as the modular architecture's boss controller and replace the modules with a linear layer decoder. • RL (Modular): For structure transfer, we train the modular architecture, discussed in Section 3.2, from scratch with RL. In task transfer, we use the modular architecture pretrained with RL on the training task and finetune both the modules and the boss controller on the test task. The inclusion of this baseline allows us to isolate the effect of the modular architecture from the pretraining and noise injection components of MeMo. • Pretrained NerveNet-Conv: We use the NerveNet network architecture proposed by [4] , consisting of an input network F in for encoding observations, a message function M , an update network U , and an output network F out for decoding. As in [4] , F in and F out are MLPs. In the convolutional [18] variant, M is the identity function and U is a weight matrix. During RL, we fix F out in a similar manner as fixing the modules in MeMo, which improves NerveNet-Conv's performance. • Pretrained NerveNet-Snowflake: Snowflake [5] is a state-of-the-art approach for training GNN policies that scale to high-dimensional continuous control. Their method involves fixing parts of the NerveNet architecture to prevent overfitting during PPO. Empirically, they find that fixing {F in , M, F out } results in the best performance on MuJoCo tasks. We follow the same parameter fixing as Snowflake. As in Snowflake, we parameterize F in and F out as MLPs and the update function U as a GRU. We use a weight matrix for M .• Pretrained MetaMorph: MetaMorph [3] is a Transformer-based approach for learning a universal controller over a large collection of robot morphologies. We adopt MetaMorph's Transformer architecture for the policy network, which adds learned positional embeddings before processing the input sequence with a Transformer encoder. As our domains lack exteroceptive observations, we directly decode Transformer encodings to controller outputs. The Transformer policy is finetuned during RL. The generalization ability of MeMo on structure transfer is shown in Fig. 6 . On all structure transfer tasks, MeMo outperforms the message-passing baselines. On the 12 leg centipede and the 10 leg hybrid, not only is MeMo 2× more sample efficient than the best baseline, but it also converges to controllers with significantly better performance than any baseline. On the 10 leg worm, MeMo outperforms all baselines in terms of training efficiency and achieves a comparable final performance as NerveNet-Conv. MeMo also outperforms all baselines on the 5 finger claw. We note that the worm transfer task is easier for GNN models, because the coordination of the shorter legs and body joints is naturally captured with multi-hop communication. MetaMorph struggles with locomotion tasks, due to the high dimensionalities of the transfer robots.The results of MeMo on task transfer are shown in Fig. 7 . As transferring from the Frozen to the Ridged, Gap, and Stepped Terrains requires the robot to overcome obstacles unseen in the Frozen Terrain, we load the pretrained boss controller and finetune MeMo end-to-end. Results (Fig. 7 ) show that on all test tasks, MeMo achieves improved training efficiency compared to MetaMorph and to pretrained MLP and modular architectures. MeMo achieves comparable performance on the Ridged and Gap Terrains and outperforms the NerveNet baselines on the Stepped Terrain, which requires the robot to climb up steps whereas the training terrain is flat. MeMo also has improved training efficiency and final performance in the grasping domain when transferring from grasping a cube to a sphere. The pretrained NerveNets struggle to coordinate the arm and claw components, resulting in high variance across different random seeds.Sum of Modularity Objectives. We answer the question of why we choose to optimize the noise injection objective rather than the sum of the modularity objectives directly. We evaluate the sum of Eq. 1 and 2 between networks trained with the noise injection objective and those trained with the sum in Table 1 . Using 100 sampled trajectories from the expert controller, we average the resulting sum of objectives over 1000 epochs, with different sampled noise on each epoch. Our results demonstrate that optimizing the noise injection objective converges to better solutions. Noise Injection Objective. The key to the success of MeMo is the introduced noise injection (NI) objective which encourages proper responsibility division among the pretrained boss controller and modules, enabling the modules to improve training efficiency when reused. We conduct an ablation study to verify this technique by experimenting on a special setting, "transferring" the controller to the same robot structure and task, a 6 leg centipede traversing a Frozen Terrain. During transfer, we reuse and freeze the pretrained modules and retrain the boss controller from scratch. With the pretrained modules from MeMo, the boss controller will be retrained much more efficiently because it only needs to take partial responsibility for the control job. We compare our method to three baselines:• MeMo (no NI): We pretrain the modular architecture end-to-end without noise injection. This ablation is equivalent to MeMo without noise injection. • MeMo (L1): During pretraining, we replace the injected noise with L1 regularization on B's output that encourages sparsity in its signal. We weigh the regularization term by a hyperparameter w and report results with the best w. • MeMo (L2): During pretraining, we replace the injected noise with L2 regularization on B's output and report results with the best weight on the regularization term. • MeMo (Jacobian): As an alternative to noise injection, we penalize the norm of the module's Jacobian using the method described in [19] .In addition, we add the training curve of RL (Modular) as a reference. The results (Fig. 8 ) show that MeMo yields a significant improvement in training efficiency over all ablations. Imitation Learning. We now answer the second question of whether imitation learning is necessary to pretrain modules with Gaussian noise injection. The results of using noise injection in reinforcement learning to pretrain modules is shown in Fig. 8 . Note that we refer to IL ablations as experiments where modules are first pretrained with imitation learning, and subsequently, the boss controller is reinitialized and retrained with RL to test the improvement in sample efficiency. RL ablations involve the second RL phase, but the pretraining stage is done with RL as well. In addition to training the modular architecture from scratch, we experiment with two methods of injecting noise during RL. The first is naive noise injection (NNI), where we inject noise into B's output when sampling rollouts and computing policy gradients. For the second, we adopt the Selective Noise Injection (SNI) technique proposed by [20] for applying regularization methods with stochasticity in RL. SNI stabilizes training by sampling rollouts deterministically and computing the policy gradient as a mixture of gradients from the deterministic and stochastic policies. However, even with SNI, the pretrained modules do not improve training efficiency.We examine how the noise injection objective forces the modules to learn a better representation of the actuator space. As discussed in Section 2, the trajectories produced by a successful policy often lie on a much lower-dimensional manifold than the actuator space. Each dimension of the manifold can be interpreted as an individual skill that the policy has learned. We can measure the dimensionality of the modules' mapping by looking at the Jacobian matrix of the worker modules with respect to the boss's signal. The trajectories outputted by a policy can likely be captured by a few dimensions of high variance corresponding to a small set of large singular values in addition to a much larger set of dimensions of lower variance corresponding to relatively small singular values. We visualize this effect by 1) computing the Jacobians at the trajectory input states of a successful policy and 2) normalizing the singular values of each Jacobian by its largest singular value and plotting the resulting values in the [0, 1] range. We expect that a module that optimizes the invariance to noise objective will have only a small number of large singular values, with the rest being close to zero. Conversely, modules that do not produce a low-dimensional manifold would have more singular values of similar magnitude, resulting in the distribution's mass clustering close to 1. We verify this intuition by sampling 100 trajectories from an expert controller for the 6 leg centipede shown in Fig. 1 .We compare the plots of the normalized singular values between MeMo and MeMo without noise injection in Fig. 9 . Without noise injection, the majority of the values are close to 1. At the other extreme, with MeMo, the values are highly clustered to the left, implying that most singular values are much smaller than the biggest singular value. We plot the singular value distributions of additional MeMo ablations in Appendix A.12. 
conclusion: 
 In this paper, we propose a modular architecture for robot controllers, in which a higher-level boss controller coordinates lower-level modules that control shared physical assemblies. We train the architecture end-to-end with noise injection, which ensures that the lower-level modules do not overrely on the boss controller's signal. In locomotion and grasping environments, we demonstrate that our pretrained modules outperform both GNN and Transformer-based methods when transferring from simple to complex morphologies and to different tasks. We ablate components of MeMo and demonstrate that the entire framework is necessary to achieve these generalization benefits.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: See Section 5.1 for details on our experimental setup and Appendix A.6 for details on hyperparameters.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: As stated in Section 5.1, we conduct compute statistics over 3 runs for locomotion and 5 runs for grasping experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Limitations and Future WorkWe now discuss the limitations of our work and potential future directions. One limitation is that our experiments were conducted using only a single type of RL algorithm (PPO) and network architecture, MLPs with Tanh nonlinearities and orthogonal initialization. Future work would involve applying and adapting our framework to different policy optimization algorithms and architectures.Because our approach involves imitation learning to pretrain the modules, we acknowledge that there is a potentially greater memory footprint for storing the imitation learning dataset.While we have demonstrated the potential of MeMo to be used for task transfer, such capabilities are inherently limited as our architecture does not explicitly encode the semantics of the task. For example, transferring the modules of a 6 leg robot walking over a Frozen Terrain to the same robot traversing a terrain with obstacles requires finetuning the architecture end-to-end, which may not always yield better results due to the instability of policy updates. Combining our framework with a mechanism for representing task semantics to enable transfer to more complex tasks is a promising direction for future work. In addition, as our experiments are only in simulation, an important line of future work is applying our approach to real world tasks.In general, we would expect our framework to face the same challenges in adapting to the real world as standard RL policies, as our problem of learning modules that generalize to different morphologies is orthogonal to learning policies that overcome the sim-to-real gap. Extending our framework to deal with challenges in sim-to-real transfer, including adapting to environmental variations, hardware inconsistencies, and discrepancies in simulated physics vs real dynamics, is an important avenue for future work.The scope of our work is limited to robots that are incrementally different from the starting robot, due to the difficulty of generalizing from a single robot and environment. One line of future work could involve adapting our training pipeline to multirobot and multitask settings, enabling our modules to capture a broader range of robot dynamics. When the dataset of robots grows larger, it can be expensive for a domain expert to manually provide labels on how a robot is decomposed into physical assemblies. However, the problem of learning reusable components bears similarity to the problem of abstraction learning studied in the programming languages community. Recent advances [21, 22] have made abstraction learning much more computationally efficient than in the past. We also see promise in adapting these techniques, which have been developed for programs, to robots that have an underlying graph structure.Here we discuss the broader social impact of our work, including its potential positive and negative aspects. On the positive side, our work enables us to train neural network architectures which are structured in a more interpretable manner, in that the modules correspond to physical components of the robot. In addition, we demonstrate generalization to robot structures and tasks that are greater in difficulty than the training setting. In summary, our modular approach is a step towards addressing the concern that neural networks are black-box models with highly limited generalization capabilities.We do not see any direct negative implications stemming from our work, as experiments are solely conducted in simulated robot environments. We note that our work does not impose safety constraints on the rollouts of the agent, which is an important limitation to address for real-world use of our method.As shown in Fig. 10 , the modular architecture starts by executing the boss controller B, which takes the full observation vector s τ as input. The full observation vector s τ includes both global observations about the agent and local observations of each actuator. The global observations consist of the agent's global position, orientation, and velocity, while the local observations are composed of joint angle, joint velocity, and local relative position and orientation in the corresponding module's frame. Given the full observation vector s τ , M outputs a latent vector H of length |P| • D, where D is the size of the embedding sent to each module.Figure 10 : Modular Architecture with Noise Injection. Our architecture consists of a higher-level boss controller B that outputs a hidden embedding, x. During imitation learning, Gaussian noise is added to x to compute H. H is split into signals that are passed into modules that output the mean of the action distribution. The dotted lines represent that in addition to H, the modules also take in subsets of the full observation vector corresponding to the state of the joints within the modules.The latent vector H is then split into |P| segments of size D and fed to modules. As shown in Fig. 11 , a module itself consists of a MLP for each actuator j n . Each MLP takes as input j n 's local features concatenated with the module's segment of the latent vector H and outputs the mean value of the action applied to j n . Our goal is to show that Eq. 3.1 is equivalent to the sum of Eq. 1 and 2 with a remaining product term. We can decompose the noise injection loss as follows:EQUATIONEQUATIONIn practice, we minimize the negative log likelihood loss during imitation learning. Similar to the above derivation, show that negative log likelihood with noise injection can be written in terms of the behavior cloning and invariance to noise losses. Let π be our modular policy, where the mean of the Gaussian distribution for each of the actions is given by W ϕ (B θ (s i )), and the standard deviation is a trainable vector, σ u . We define the behavior cloning and invariance to noise losses as negative log likelihoods in Eq. 11 and 13 respectively.EQUATIONEQUATIONp π (W ϕ (B θ (s i )) | s i , η) = 1 σ u √ 2π exp - 1 2σ 2 u (W ϕ (B θ (s i ) + η) -W ϕ (B θ (s i ))) 2 (12) -E η [E i [log p π (W ϕ (B θ (s i )) | s i , η)]] = E η E i log(σ u √ 2π) + 1 2σ 2 u (W ϕ (B θ (s i ) + η) -W ϕ (B θ (s i ))) 2 (13)We now consider the conditional log likelihood with noise injection and show how it can be decomposed in terms of the above losses.EQUATION)where C is the product term.Let D be the base hidden size of the network. As typical in PPO, we use Tanh nonlinearities and orthogonal initialization for the standard MLP and modular architectures. The standard MLP and boss controller are 2 layer neural networks. The size of the first layer is D while the second layer has L • D hidden units, where L is the number of modules. The standard MLP also has a final linear layer to decode the actions. For all policy architecture variants, the value function is defined as a 2 layer neural network with D hidden units each, followed by a linear layer.In PPO, agents iteratively sample trajectories based on the current policy and subsequently perform optimization on a surrogate objective that first-order approximates the natural gradient. The surrogate objective prevents unstable updates to the policy by clipping the probability ratio r τ (θ;θ old ) = π θ (a τ | s τ )/π θ old (a τ | s τ ).Optimizing the clipped objective is done with the policy gradient [23] .The RL loss that all architectures optimize includes the surrogate objective, a weighted value function loss, and a weighted entropy bonus to encourage exploration:EQUATION)where Âτ is the generalized advantage estimation (GAE) [24] . ϵ is the clip value, c 1 is the weight on the value function, and c 2 is used to balance the entropy bonus.We For experiments with pretrained policy weights, we initialize the learned logstd to -1.0. For all models on the locomotion tasks, we perform a secondary search over the batch size in [256, 512, 1024, 2048] until performance decays. We find that MeMo works best with smaller batch sizes: 256 on the 12 leg centipede, 512 on the 10 leg worm, and 1024 on the 10 leg hybrid. The modular architecture also sees an improvement when using 1024 for all three robots, whereas the default batch size of 2048 works best for the MLP architecture. NerveNet-Conv and NerveNet-Snowflake improve with a batch size of 1024 on the 12 leg centipede and 10 leg worm. We do not see a significant improvement when decreasing the batch size for MetaMorph. Due to the higher variance in reward for grasping tasks, we keep the same batch size for the transfer experiments.For the modular architectures, we parameterize each joint network within the modules with a 2 layer MLP with 32 hidden units per layer. For imitation learning, we use a batch size of 1024 and tune the learning rate in [7e-4, 1e-3, 2e-3, 4e-3, 7e-3]. All models are trained for 175 iterations of DAgger. We sample 500 trajectories from the expert controllers of the 6 leg centipede, 6 leg worm, and 6 leg hybrid and 250 trajectories from the controller of the 4 finger claw as the validation sets. We make sure that the architectures pretrained with imitation learning achieve a comparable average reward as the expert controller when a number of trajectories are sampled from them.For NerveNet, the input network is a single layer with size D followed by a Tanh nonlinearity. We have a separate output network for each joint, and each output network is a 2 layer MLP with 32 units per layer. Table 3 summarizes the hyperparameter search that we perform for NerveNet. We perform grid search over the number of layers and the size of the messages passed by the propagation network. We choose the smallest architecture size that achieves a similar average reward as MeMo. Adding a skip connection from the root to all joints improves NerveNet's validation score in imitation learning and enables the use of smaller architectures that do not overfit as easily. For MetaMorph, we tune the number of attention layers in [2, 3, 4] and otherwise use the same architecture hyperparameters as [3] , listed in Table 4 . For a fair comparison, we use a MLP critic with the Transformer policy during RL rather than a Transformer critic, as the critic is trained from scratch during transfer. We find that positional encoding improves imitation learning pretraining, enabling the use of smaller architectures. For each token that corresponds to a joint, we include the one-hot encoding of the joint type in the observations. We keep a running mean and variance to normalize the state space. Relative positions / orientations are relative to a joint in the same module as a given joint. For grasping, we use relative joint orientations as global joint orientations depend significantly on how high the claw is lifted. Table 5 and 6 detail the observation space in locomotion and grasping. For locomotion, "base" refers to the forwardmost wide body segment of the robot. As the joints are hinge joints, they only have one degree of freedom.The token type refers to the observation processing for MetaMorph -each input sequence consists of a single "global" token with the corresponding global observations for the robot concatenated with zero padding for the local observations, and the rest of the tokens are "joint" tokens with zero padding for the global observations concatenated with the local observations of the corresponding joint. We choose to integrate global information at the encoder level rather than the decoder level as our global features are low-dimensional: only 16 dimensions at most. The original MetaMorph architecture considers exteroceptive features from camera or depth sensors as global, which are much higher dimensional and are concatenated at the decoder level to prevent the dilution of local proprioceptive information.We run experiments on 2 different machines with AMD Ryzen Threadripper PRO 3995WX processors and NVIDIA RTX A6000 GPUs. Both machines have 64 CPU cores and 128 threads. The main cost of running the agent in both the RoboGrammar and the DiffRedMax environments is the cost of simulation, which is CPU-intensive. For the MLP-based architectures, we only use CPU cores for computing rollouts in parallel environments via vectorization and backpropagating the policy gradient. For NerveNet, in the locomotion domain, we find it helpful to vectorize environments while performing backpropagation with a GPU. For example, the RL stage of MeMo on the 6 leg centipede takes less than a day to complete, whereas training a 3 layer NerveNet-Conv with the same number of processes and batch size requires 3-4 days without a GPU. We note that our resources are shared, and the wallclock time varies depending on the other processes running on the same server.We provide more details on the RoboGrammar tasks. On all locomotion tasks, the maximum episode length is 128. Full details of the environments can be found in the RoboGrammar codebase [16] . • Frozen Terrain: A flat surface with a friction coefficient of 0.05.• Ridged Terrain: Ridges are placed an average of one meter apart across the width of the terrain.• Gap Terrain: A series of platforms separated by gaps.• Stepped Terrain: A series of steps with varying height, resembling a flight of stairs.For all RoboGrammar locomotion environments, the reward at timestep τ is the sum of the rewards at each sub-step t. The training reward function at substep t isEQUATION)where N is the dimension of the action vector, and each dimension is normalized to [-1, 1]. The first two terms encourage high velocity in the x-direction and maintaining the robot's initial orientation respectively. The last term is a regularization penalty to reduce the variance across different runs.The reported reward curves do not include the regularization penalty.For grasping, the goal is to grasp an object and lift it as high as possible and the maximum episode length is 50. As in prior work [6] , we follow the convention of controlling the actuators with relative positions rather than absolute positions. The reward at timestep τ is the sum of the rewards at each sub-step t. The full set of parameters used to construct the DiffRedMax simulation will be released with our source code. Below is the reward function used, where object z refers to the object's z-coordinate and avg_fingertip_dist is the mean distance of the claw's fingertips to the object's surface. We approximate the cube's surface with the surface of the largest sphere that fits in the cube. all_fingers_in_contact checks whether or not all fingers of the claw is within a small distance from the surface of the object.R(s t , a t ) = 10 • object z -0.1 • avg_fingertip_dist all_fingers_in_contact -0.1 • avg_fingertip_dist !all_fingers_in_contactThe penalty on avg_fingertip_dist encourages the fingers to grasp the object. We only include the reward term on object z when all_fingers_in_contact is satisfied in order to prevent the claw from throwing the object.A.10 SourcesWe use the PPO implementation provided in https://github.com/ikostrikov/ pytorch-a2c-ppo-acktr-gail (MIT License). Our NerveNet implementation is adapted from a PyTorch version of the original NerveNet codebase: https://github.com/HannesStark/ gnn-reinforcement-learning. We adapt our MetaMorph implementation from the official codebase: https://github.com/agrimgupta92/metamorph/tree/main. We use the official RoboGrammar [16] (MIT License) and DiffRedMax [17] (MIT License) simulators.Noise Injection. One line of work focuses on explaining the generalization benefits by noise injection by deriving explicit regularization terms. [25] studies Gaussian noise injected into network activations at each layer and derive an explicit regularization term by marginalizing out the noise. [26] analyzes the effect of Gaussian noise injection to the training data and find that the effect is equal to weighted ridge regularization as the number of noise injections approaches infinity. [27] study injecting noise into RNN hidden states and identify an explicit regularizer for small noise variances.Hierarchical and Multi-Task RL. Our proposed modular architecture bears similarity to those used in hierarchical RL [28, 14] . However, a key difference is that our architecture is hierarchical with respect to the morphology of the robot, not the temporal structure of the task. To train robots that perform a diverse set of skills and generalize to new tasks, prior work leverages the shared structure of tasks, such as through graph representations [29, 30] that represent task compositionality, or through language representations [31, 32] . While many works in MTRL focus on a single morphology, recent efforts [33] have proposed representing both morphology and task in a single graph, enabling architectures trained on this unified IO representation to transfer to unseen morphologies and tasks.Multi-Robot Coordination. Past works in multi-robot coordination bear similarity to our work in either the modularity of the architecture or the learning of a higher-level coordination mechanism, analogous to our boss controller, between different agents. In particular [34] uses a modular architecture, in which a higher-level meta-policy coordinates various skills with a behavior embedding. [35] proposes to learn a useful latent action space for coordinating a multiagent system via an information bottleneck. The information bottleneck helps in learning a latent action space from the full set of observations that is useful in coordinating decentralized agents at inference time.In Fig. 12 , we test the zero-shot generalization of the pretrained NerveNet-Conv baseline by fixing all of its weights and only training the learned standard deviation when transferring from the 6 to the 12 leg centipede. Its poor performance demonstrates the difficulty of the transfer task, in spite of the physical similarities between the 6 and the 12 leg centipede.In addition, we demonstrate the capability of MeMo is scaling to much higher dimensionalities by transferring from a 6 to a 24 leg centipede (Fig. 13 ) on Frozen Terrain. MeMo significantly outperforms RL (Modular), the strongest baseline on the 6 to 12 leg centipede transfer task.In Fig. 14 , we show that our framework has the potential to correct the dynamics of the system during transfer, specifically when some joints in the transfer robot fail to perform as expected even when given the correct control signal. In the 6 to 12 leg centipede transfer, we randomly select 7 out of 70 joints in the 12 leg centipede to be uncontrollable (for each seed of the experiment, a different subset of uncontrollable joints is sampled, and we have 3 random seeds). For the uncontrollable joints, we pass in a small random noise instead of the controller's output to the simulator. MeMo significantly outperforms the RL (Modular) baseline, achieving its final reward in less than half of the timesteps.In Fig. 15 , we test the transfer capabilities of MeMo to morphologies smaller than the starting structure. Specifically, we transfer modules from a 6 leg to a 4 leg centipede on the Frozen Terrain and from a 4 finger to a 3 finger claw grasping a cube. MeMo achieves improved training efficiency to policies trained from scratch and performs similarly to the strongest baseline in each domain.In Fig. 16 , we compare the singular value distributions of MeMo to the ablations described in Section 5.3 and MeMo with a lower value of σ. In Fig. 17 , we run MeMo on the 6 to 12 leg centipede transfer where either the boss controller or the modules are 4 layer MLPs instead of 2. Both of these variants perform similarly to the original architecture. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Correspondence to megantj@mit.edu. Code can be found at https://github.com/MeganTj/MeMo. 38th Conference on Neural Information Processing Systems (NeurIPS2024). 


Decision: The paper tackles an interesting and challenging problem. The reviewers appreciated the replies, additional details, and additional experiments. Overall the proposed method is an important first step, however there is also quite some way to go to make it more general (e.g. for more complex systems, multi-agent setting). On the flip-side focusing on simpler systems allows for an in-depth analysis.
After the discussion all reviewers are in favor of accepting this paper.

Reviews: review: paper_summary: paper_summary: This paper presents a method for learning modular controller that can be transferred and adapted to different morphology of robots and tasks. The high level idea is to decompose control of each motor through distilling a learned hierarchical RL policy and then uses these primitive policies as building blocks to additional motors added.
strength_weakness: strength_weakness: The proposed method only trains the master policy at adaptation stage, which assumes the change of morphology does not induce change in dynamics of the system such that drastically different low-level policies will be needed to solve the task.
questions: questions: Will this method generalize to multi-agent RL setting where the modular policy might be more complex than controlling 1-DoF motor?
limitations: limitations: see weakness
paper_summary: paper_summary: This paper introduces a new framework designed to create modular controllers allowing for quicker adaptation of control strategies when building new robots with similar methodology. The MeMo framework employs a novel modularity objective optimized alongside standard behavior cloning loss through noise injection. Experiment results in locomotion and grasping environments, ranging from simple to complex robot morphologies, demonstrate that MeMo improves training efficiency compared to multiple baselines. Additionally, MeMo’s modular approach enhances both structure and task transfer capabilities.
strength_weakness: strength_weakness: * The tasks evaluated are relatively simple, only basic locomotion tasks and quite simple manipulation tasks are considered. 
* All tasks involve only a rigid body, which could be simulated much faster with GPU with the recent simulator, which should deliver a much shorter training time compared to the reported time in the appendix. It’s a bit questionable whether the proposed method is necessary if training a new policy from scratch takes a short period.
questions: questions: No specific questions to add.
limitations: limitations: This work has provided reasonable discussion over the limitation and no potential negative social impact of the work needs to be considered.
paper_summary: paper_summary: The MeMo framework presented in this paper proposes an innovative approach for enhancing the transferability of control systems across robots with varied morphologies by utilizing pre-trained, modular controllers. This method facilitates rapid adaptation to new robot designs by leveraging previously trained modules, thereby significantly improving training efficiency over conventional methods like graph neural networks and Transformers.
strength_weakness: strength_weakness: N/A
questions: questions: 1. The comparative analysis raises some questions regarding the fairness of the evaluation. Specifically, were the control modules of the compared methods also retrained on your newly designed robots, or was it only the master controller that was retrained? If the latter is the case, this might lead to an unfair comparison since these pre-trained models have not been directly exposed to the newly designed robot configurations prior to testing.
2. The transition from simulation to real-world applications is not addressed. The paper would benefit from a discussion on the expected challenges when implementing these modules in actual robotic systems, such as dealing with hardware inconsistencies and environmental variations. For example, how does the framework handle discrepancies in the physical properties between different robot assemblies, such as variations in leg mechanics?
limitations: limitations: N/A
paper_summary: paper_summary: This manuscript presents a hierarchical controller for reinforcement learning-based control. In particular, given a robot that can be decoupled into a high-level controller and a few low level (joint level) controllers, the two modules are learned jointly via a behavior cloning objective. The proposed method further makes the high-level controller robust to noise from the output of the low-level controllers by adding noise to their outputs. The model experiments on a few morphologies (6/12-leg centipede, 6/10-leg worm, etc) to demonstrate its effectiveness over vanilla RL and NerveNet.
strength_weakness: strength_weakness: 1. Opposing strength 2, especially in simulation, many existing works randomize over many morphologies to enable efficient transfer between them or to an unseen morphology. It is unsure how this method can be trained on a dataset that contains more than one morphology. 
2. For the experiments, the reviewer is uncertain how significant is the morphology change in affecting the dynamics of the system.
questions: questions: 1. Can the expert policy on 6-leg centipede, and 6-leg worm be executed on 12-leg centipede and the 10-leg worm? The same question applies to the manipulation setting (4-finger to 5-finger claw).
2. How effective is the method at correcting the dynamics of the system? I.e. after pre-training on the 6-leg centipede, if two or more of the pre-trained joints failed (i.e. uncontrollable), how fast can it adapt to the new setting?
limitations: limitations: Present in the manuscript.

Comments: title: Official Comment by Submission3440 Authors
thanks for clarification
Thank you for your response
Thank authors for detailed explanations
comment: > Can you clarify how you may train the modules across different morphologies?

Consider an example where the training morphologies consist of the 4 leg centipede and 6 leg centipede, both of which are composed of “leg” and “body” modules, denoted as $\textbf{W}_0$ and $\textbf{W}_1$ respectively. Then the modular architecture for the 4 leg centipede is a composition of the boss $\textbf{B}_1$ and its set of modules $\mathcal{W}_1 =\\{\textbf{W}_0^0, …, \textbf{W}_0^3, \textbf{W}_1^0 \\}$ (4 leg modules and 1 body module). For each module, the subscript corresponds to module type and the superscript denotes different instances of the same module type, which share model parameters. Similarly, the 6 leg centipede is a composition of the boss $\textbf{B}_2$ and its set of modules $\mathcal{W}_2 = \\{\textbf{W}_0^4, …, \textbf{W}_0^9, \textbf{W}_1^1, \textbf{W}_1^2\\}$ (additional 6 leg modules and 2 body modules). While the boss controllers differ across various morphologies, the modules are shared. We can then train both architectures end-to-end with imitation learning while injecting independent noise vectors $\eta_1, \eta_2$ to the output of $\textbf{B}_1$, $\textbf{B}_2$ respectively at each batch. 

> I am referring to the second setting mentioned in the response. It would be great if an experiment could be conducted in such a setting, so that we can see the trained policy is robust to change in dynamics..

Thanks for the clarification. We have run this experiment on the 6 leg to 12 leg centipede transfer task, where during transfer, we randomly select 7 out of 70 joints in the 12 leg centipede to be uncontrollable (for each seed of the experiment, a different subset of uncontrollable joints is sampled, and we have 3 random seeds). For the uncontrollable joints, we pass in a small random noise instead of the controller's output to the simulator. Below, we compare MeMo’s performance on this transfer task to RL (Modular), the strongest baseline from the 6 to 12 leg centipede transfer. MeMo significantly outperforms the baseline, achieving its final reward in less than half of the timesteps. This demonstrates the potential of our framework to adapt to unforeseen dynamics at transfer time. 

| |4e+6 timesteps | 8e+6 timesteps|
| :---------------- | :------: | ----: |
|RL (Modular)| 1203|1485 |
|MeMo|**1591** |**1793** |

Edit: Added table with final rewards.
Thank the authors for addressing my concerns, I would like to keep my original evaluation.
Dear authors, 

Thanks for addressing my concerns regarding your paper and sorry for the belated review. 

> On the one hand, if you do have many morphologies, you can easily extend our approach to train the modules across the different morphologies and you may even get better modules

Can you clarify how you may train the modules across different morphologies? 

> How effective is the method at correcting the dynamics of the system? I.e. after pre-training on the 6-leg centipede, if two or more of the pre-trained joints failed (i.e. uncontrollable), how fast can it adapt to the new setting?

I am referring to the second setting mentioned in the response. It would be great if an experiment could be conducted in such a setting, so that we can see the trained policy is robust to change in dynamics..  

For my final rating, I will take into consideration the arguments which arise during the upcoming AC-Reviewers discussion phase. Overall, I lean towards increasing my score.

Best regards!
I have read the authors' response and appreciate the clarification. It would be interesting to see extending this method to multi-agent RL settings!
Thank you very much -- we're happy we were able to address your questions!
Thank authors for detailed explanations. I don't have any other questions and am happy to raise my points to 6.