MeMo: Meaningful, Modular Controllers via Noise Injection
Authors: Megan Tjandrasuwita
Jie Xu
Armando Solar-Lezama
Wojciech Matusik
Venue: NeurIPS 2024 poster (2024)
Keywords: modular neural network policy
policy transfer
imitation learning
reinforcement learning
Primary area: robotics
TL;DR: We present a method for pretraining modular controllers from a single robot and environment that significantly speeds up RL training for locomotion and grasping when reused on more complex morphologies.

Abstract: Robots are often built from standardized assemblies, (e.g. arms, legs, or fingers), but each robot must be trained from scratch to control all the actuators of all the parts together. In this paper we demonstrate a new approach that takes a single robot and its controller as input and produces a set of modular controllers for each of these assemblies such that when a new robot is built from the same parts, its control can be quickly learned by reusing the modular controllers. We achieve this with a framework called MeMo which learns (Me)aningful, (Mo)dular controllers. Specifically, we propose a novel modularity objective to learn an appropriate division of labor among the modules. We demonstrate that this objective can be optimized simultaneously with standard behavior cloning loss via noise injection. We benchmark our framework in locomotion and grasping environments on simple to complex robot morphology transfer. We also show that the modules help in task transfer. On both structure and task transfer, MeMo achieves improved training efficiency to graph neural network and Transformer baselines.

MeMo: Meaningful, Modular Controllers via Noise Injection
Megan Tjandrasuwita MIT megantj @eitttte@mim..m edu
Jie Xu NVIDIA jiex@nvidia. com
Armando Solar-Lezama
Wojciech Matusik MIT vojciechemit.edd
MIT asolar@csail.mi edu
Abstract
Robots are often built from standardized assemblies, (e.g. arms, legs, or fingers), but each robot must be trained from scratch to control all the actuators of all the parts together. this paper we demonstrate a new approach that takes single robot and its controller input and produces set of modular controllers for each of these assemblies such that when a new robot is built from the same parts, its control can be quickly learned by reusing the modular controllers. We achieve this with a framework called MeMo which learns (Me)aningful, (Mo)dular controllers. Specifically, propose novel modularity objective to learn appropriate division of labor among the modules. We demonstrate that this objective can be optimized simultaneously with standard behavior cloning loss via noise injection. We benchmark our framework in locomotion and grasping environments on simple complex robot morphology transfer. We also show that the modules help in task transfer. On both structure and task transfer, MeMo achieves improved training efficiency graph neural network and Transformer baselines
1 Introduction
Consider the following scenario: A roboticist is designing a robot with legs, such as the one seen in the left image of Fig. [ and has trained standard neural network controller with deep reinforcement learning (RL) to control the actuators circled in green. However, after more testing, they realize that the design of the robot needs to be extended with another pair of legs to support the desired amount of weight. Even though the new leg robot is still composed of the same standard assemblies, the roboticist is unable to reuse any part of the 6 leg robot's controller. While many works 123 have studied structure transfer, or transferring neural network controllers to different robot morphologies, these works take purely data-driven approach of training universal controller dataset representative of the diversity and complexity of robots seen in testing. contrast, we desire learn transferable controllers from only single robot and environment, obviating the requirement for substantial training dataset and resources to perform multi-task RI. Our experiments demonstrate that state-of-the-art approaches for transferring control to environments with incompatible state-action spaces struggle to generalize in this highly data-scarce setting.
Motivated by the above scenario, we propose framework, MeMo, for pretraining (Me)aningful (Mo)dular controllers that enable transfer from single robot to variants with different dimensionali- ties. Learning transferable modules from single robot trained single task is challenging, even
CCorrespondence megant @mit
fomit. edu. Code be found at htapa:///gtthubb.cmm//eggggggg
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

W
33
29
25
1B
18
24
28
30
32
31
12
28
30
W3
27
17
4
16
26
W3
43
12
16
26
15
10
14
20
Wi
W
0
W2
root
Figure I: Graph Structure and Neural Network Modules of the 6 Leg Centipede. Left: The robot's joints are labeled numerically and circled. Right: The joints form the nodes and the links are the edges. The subset of joints that form each leg module are circled in red, while those that comprise each body module are circled in blue. Neural network modules are denoted as W where k refers to the type, e.g. all leg modules type O, and denotes different instances of the same module type.
when we focus on transfer among robots with similar global morphologies. The key insight MeMo leverages is that robot built from assemblies of individual components, such as the leg of a walking robot or the arm of a claw robot. These assemblies are specified by a domain expert who is able to account for the constraints imposed by the robot's hardware implementation in their specification. Given this information, MeMo learns assembly-specifee controllers, or modules, responsible for coordinating the individual actuators that comprise a given assembly, which are coordinated by higher-level boss controller. As we are able to reuse the modules when transferring to robot built from the same assemblies, the problem of learning a controller for a different morphology boils down to learning the coordination mechanics among assemblies, rather than having to coordinate at the granular level of individual joints. Returning to the 6 leg robot in Fig. [ we see that the robot is comprised of multiple "leg" and "body" assemblies, circled in red and blue respectively in the right image. Module parameters are shared between assemblies the same type, providing multiple training instances that help our modules generalize. After training the modules with MeMo, the "leg" and the "body" modules can then reused to speed up the training of different robot's 's controller, such an 8 leg robot.
To achieve this improved training efficiency, a key challenge is balance the labor between the boss controller and the modules. In one direction, to prevent the modules from becoming too robot- specialized, we introduce information asymmetry into our architecture, where the modules are limited to seeing the local observations of the actuators that belong in the module. In the other direction, controlling the assembly through the module must be simpler than controlling the assembly directly, since otherwise there is no benefit to this new architecture. This is achieved by new modularity objective (Section 2) that forces the modules to capture as much of the coordination mechanics within a subassembly possible, given limited local observations. In practice, we use noise injection (Section 3.I to optimize the new objective simultaneously with standard behavior cloning loss.
To evaluate the transferability of the learned module, we apply MeMo in locomotion and grasping domains. We design two types of transfer: generalizing to more complex robot structures and to different tasks. When transferring model weights from a simpler agent, we show that MeMo significantly improves the sample efficiency of performing RL on more complex agents. We compare our framework with NerveNet, an alternative approach for one-shot structure transfer 4 5) and MetaMorph R3 an approach for learning universal controllers. Our experiments show that MeMo either exceeds or matches NerveNet and MetaMorph's 's training efficiency during transfer, as the message-passing policies are prone to overfitting during pretraining.
2 Motivation for Modularity Objectives
Our goal to maximize the extent to which the assembly-specific modules take responsibility for the behavior of the robot. In this section, we formalize the objectives that our training pipeline should achieve. Given an expert controller F, one can train a modular controller, consisting of a higher-level B(oss) module that sends signal to each of the W(orker) modules, to mimic F's behavior using the

---

standard behavior cloning objective. Let B be parameterized by 0 and W be parameterized by Φ. For simplicity, this section assumes that there a single W module.
Definition 2.1. Behavior Cloning Objective. For system with states 81 S, the modular policy whose output is W (B)( imitates the expert monolithic policy F.
argming E, (W (Ba(s.)) - F(+:))
(I)
The behavior cloning objective ensures that the composition of modules can perform the desired task, but it is not enough to ensure that the worker module is useful. In software engineering, component most useful if it can provide a narrow interface to rich set of functionality. In the context of modularity, this is analogous to W giving B only few degrees of freedom to control the system's outputs; otherwise, a module W that gives B full control over the actuators will leave W with no real responsibility for the robot's behavior. Then, when W used with new robot or for slightly different task, B needs to relearn all the details of how to control the output for that new setting.
For example, consider robot arm with 5 degrees of freedom that controls lever in Fig. 2 An ideal worker module would take as input a signal corresponding to the desired angle the lever and would be responsible for coordinating the signals to the five actuators to achieve the lever's desired position. This would mean that if we want to reuse W in controlling two arms, the new boss B will only have to learn how coordinate the two angles, and not all 10 actuators.
Goel: Push lover 90' dockwise (e(
W Action Space
State at (2%,
Curput Sipnce
135
W Astion Space
WIll out
Figure 2: Effect Modularity Objectives. Consider a module with actuators, denoted in orange, trained to push a lever clockwise. As the state of the lever is a function of its angle 0, a module trained by MeMo represents the control signals as a one-dimensional manifold with respect to B's signal. When noise is added to B's signal, the outputted actions remain the manifold. Without MeMo, perturbations to B's signals cause deviations from the high reward trajectory.
In practice, though, foreing the interface between B and W to be a one dimensional vector makes the optimization problem very difficult. Instead, our approach will be to use a larger vector as the interface between the two modules, but introduce an additional optimization objective. Intuitively, the interface is effectively narrow when B's s signal can be decomposed into a small set of dimensions that result greater variance in W's output and much larger set of dimensions that do not cause significant perturbations in W's output. In other words, the null space of W's W's Jacobian would be angher-dimensione meaning that W has a greater tolerance for error in B's signal that fall in the directions of the null space. To encourage W to be less sensitive to perturbations in B's signal, we minimize the distance between W (B,(s)) + 17). where 1j is noise vector, and W (B:)..).
Definition 2.2. Invariance to Noise Objective. Let 7) be noise vector. The difference between the result of applying W on the distorted input and on the undistorted imput is D(s,, 7) = (B9(s); - (s:) is added on each epoch, we average the difference 7) W (s,)).)) As new distortion Bp(s;) over the added noise.
argmin,,0 o E, E, [(s 1
(2)
In practice, we sample 17 from a Gaussian distribution with E[y] 0 and E[n n)
I where
1.0. In Section 5.3 we demonstrate that invariance to noise objective is the critical component in our framework that yields positive transfer benefits. In Section [.4] we show that optimizing the noise invariance objective reduces the effective dimensionality of B's signal.
3

---

3 Method
We describe our approach MeMo, an algorithm for learning reusable control modules. Section 3.1 we show that the modularity objectives can be optimized with noise injection. In Section 3.2 extend our formulation to systems with more than one module and detail our training pipeline.
we
3.1 Objective
We propose optimize both modularity objectives simultaneously with noise injection. Definition 3.1. Noise Injection Objective. Here, can be viewed as "injected noise."
argming, g.6 En E, (Wa(BBos)) + +n) - F(s,))²]
(3)
The noise injection loss can be decomposed as follows:
L = E. [W ((Wa(Bo(s)) F(s.)) + E. TE [D(s "]]
+ E, (2 (Bo - F(s,)) [D(s,, 7)]l
(4)
See Appendix A.4 for the derivation of the decomposition. Without the last term, which call the product term, noise injection is equivalent to the sum of Eq. [ and Analyzing the product term further, by the Mean Value Theorem, D(s;, 11) W (Ba(s): + 77) W (s:)) = Jw(=)Tn) for
E L, where Jw denotes the Jacobian of W with respect to B's output and L is the line segment with (s;) and (s:) + 17 as endpoints. Applying the expectation over the noise:
E,, [D n) E,, [=(
Note that depends on the value of so it cannot be pulled out of the expectation. However, in prac- tice. we expect that Jw (2)T s Jw Jw(Ba(6 (s.)) This implies that E,, [D(,,, 7)] = Jw (Ba TE) [n] O, making the product term negligible, Empirically, in Fig. 3 we show that this product term indeed becomes much smaller than the sum of the two modularity objectives as training proceeds.
Centipede
Hybrid
Claw
ratlo
0.
0.
3.0
0.5
0.
0.2
0.2
# Training Updates
Figure 3: Noise Injection Error. Over the course of training, we compute ratio = /cpl/cC + L2) where |2pl is the magnitude of the mean product term over the minibatch and L1 and L2 are the mean behavior cloning and invariance to noise losses, We compute training statistics over 5 runs and indicate standard deviation by shaded areas. (Righ)) For all starting morphologies, the modularity objectives dominate the loss as the ratio is less than for all updates.
3.2 Modular Architecture and Training Pipeline
Modular Architecture. Although thus far we have only a single module W, robot is often comprised of multiple modules controlling physical assemblies that are common among different morphologies. Formally, we assume that we are given partitioning P of an agent's joints Jo....N-1 We design a modular policy composed of a boss controller B that outputs intermediate signals to neural network modules that decode actions. Each element of the partition, e.g. a subset of actuators, is a module instance of type l., which we denote as W In total, there are [P] modules. Modules of the same type share the module parameters, yet each instance will receive different message from B. We detail our architecture further in Appendix A.3

---

Training Pipeline. To train our modules, inspired by previous works that combine RL and IL 6 7 g we first train F using RL. During the RL stage, we use proximal-polics optimization to train actor-critic controllers. The critic is a MIP, whereas the actor is standard ML. when training the expert controller and modular architecture when transferring pretrained modules. Once trained, we train modular policy (a, 84) with II via DAgger [T0], with noise injected into Bg 's output. At each iteration & of DAgger, we sample a trajectory Da: from ne.6- provides the correct action to each E Dr. and D is aggregated into the full dataset D {(s;, a.)). To optimize the objective defined in Section 3.1 we minimize = -E [log a ( 8,)]. We derive decomposition of
the negative log likelihood loss with noise injection into the modularity objectives in Appendix AA.5 After transferring the modules to a new structure or task, we perform RL to retrain B or finetune the architecture end-to-end, Our pipeline is summarized in Fig. 4 Appendix A.6ddetaill our RL and IL hyperparameter settings.
Phase 1: Reinforcement Learning
Phase 2: Imitation Learning
Phase 3: Transfer Learning
Expart
Student
Injection
We
W:
We
W.
Figure 4: Training Pipeline Overview. In Phase 1, first train expert controller for the training robot using RL. In Phase 2, we pretrain modules with noise injection during imitation learning. In Phase 3, we transfer the modules to a different context and retrain the boss controller B3.
Related Work
Our work relates to modular controllers and structure transfer. Related works in noise injection, multi-robot RL, and hierarchical RL are discussed further in Appendix [A.1]]
Modular Controllers. Our work relates to prior works that train modular policies for robot designs.
[T learns neural network policies that are decomposed into "task-specific" and "robot-specific" modules and performs zero-shot transfer to unseen task and robot-specific module combinations. [I] coordinates modular policies shared among all actuators via message passing. [2] uses GNN to
internally coordinate between spre-specific nodes with shared module parameters between nodes corresponding to the same part. [13] proposes the Dynamic Graph Network to control self-assembling agents, consisting of modules that are shared across agents.
Structure Transfer. In the hierarchical RL setting, [44 uses imitation learning to train policies that represent long-horizon behavior and improve sample efficiency when transferred from simple to complex agents. [15] transfers policies to robots with significantly different kinematics and morphology by defining continuous evolution from the source to the target robot. Previous works use message-passing policy architectures to generalize across morphologies II 5 In the multi- task setting [2] proposes Transformers as policy representations that remove the need for multi hop communication. [3] scales Transformer- based policies to larger and more diverse datasets robots.
Experiments
With our experiments, seek to answer four questions. 1) Do the modules produced by McMo generalize when transferred to different robot morphologies and tasks? 2) When pretraining modular controllers with imitation learning, does the Gaussian noise injection help? 3) In the pretraining phase, why do we use imitation learning rather than injecting noise in reinforcement learning? 4) How does modularity objective yield better representations of the actuator space? We answer question 1) in Sections 5.1 and 5.2 2) and 3) in Section and 4) in Section 5.4
5.1 Transfer Learning
We benchmark our framework on two types of transfer: structure and task transfer. While our framework is designed primarily for structure transfer, we use task transfer experiments as an additional means of evaluating the quality of the learned representations. For the locomotion
5

---

experiments, we perform experiments on the tasks introduced in RoboGrammar [16] with training statistics computed as the average reward across 3 runs, with standard deviations indicated by shaded areas. For the grasping domain, we construct abject-grasping tasks using the DiffRedMax simulator 17 and compute training statistics as the average reward across runs. Additional details on the reward functions used are in Appendix A.9 We visualize train and test robot morphologies for structure transfer in Fig. 5 and the train and test tasks for task transfer in Fig. 18
Locomotion. We design three structure transfer tasks in the locomotion domain, in which the goal is to move as far as possible while maintaining the robot's initial orientation. The starting morphologies are the 6 leg centipede robot, the 6 leg worm robot, and the 6 leg hybrid. The 6 to 12 leg centipede transfer demonstrates scalability to transfer robots with many more modules than seen in training. The 6 to 10 leg worm shows that MeMo generalizes with only 1-2 instances of the same module seen in training. The 6 and 10 leg hybrid robots involve three types of modules, demonstrating scalability to more complex training robots. For task transfer, we transfer policy weights pretrained on 6 leg centipede locomoting over the Frozen Terrain to three terrains that feature obstacles or climbing.
Grasping. In grasping, the goal is to grasp and lift an object as high as possible. We design a grasping robot consisting of an arm that lifts claw grasping cube. The structure transfer is from a 4 finger to a 5 finger claw. For task transfer, we transfer policies trained to control the 4 finger claw grasping a cube to the same robot grasping a sphere of similar size and weight.
to 12 Lag Centipade
to 10 Lag Worm
to 10 Lag Hybrid
to 5 Finger Clew
W
WWW
W WWW
W
Figure 5: Structure Transfer Tasks. Left: Transfer "leg" and "body" modules from a 6 to a 12 leg centipede. Left Middle: Transfer "body" and "head" modules from a to a 10 leg worm. Right Middle: Transfer "leg." "head," and "body" modules from a 6 to a 10 leg hybrid. Right: Transfer "arm" " and "finger" modules from a 4 to a 5 finger claw.
Baselines. We compare MeMo MLP and modular policies trained from scratch well pretrained NerveNet [4] and MetaMorph (3 baselines. NerveNet takes as input the underlying graph structure of the agent, where the nodes are actuators and edges are body parts. The graph structures of the train morphologies are detailed in Appendix A3 For MetaMorph, a Transformer-based approach, we convert the global observations and local observations for each actuator ID sequence of tokens. Full training details and state space descriptions are included in Appendix An6 A.7 respectively.
For structure transfer, due to the change in the observation space, we train 2 layer MLP policy from scratch with RL. In task transfer, we use a MLP pretrained with RL on the original task and finetune it on the test task. For a fair comparison, we use the same architecture size the modular architecture's 's boss controller and replace the modules with a linear layer decoder.
RL (MLP):
RL (Modular): For structure transfer, we train the modular architecture, discussed in Section 3.2 from scratch with RL. In task transfer, we use the modular architecture pretrained with RL on the training task and finetune both the modules and the boss controller the test task. The inclusion of this baseline allows us to isolate the effect of the modular architecture from the pretraining and noise injection components of MeMo.
Pretrained NerveNet-Cony: We use the NerveNet network architecture proposed by 4 consisting of an input network Fi, for encoding observations, a message function M, an update network U, and an output network Fout for decoding. As in [4], Fi, and Fout are MLPs. In the convolutional [8] variant, M is the identity function and U is a weight matrix. During RL, we fix Fout in similar manner as fixing the modules in MeMo, which improves NerveNet- Conv's performance. approach for training GNN policies that scale to digh-dimensional continuous control. Their method involves fixing parts of the NerveNet architecture to prevent overfitting during PPO. Empirically, they find that fixing (Fin, M, Fout) results in the best performance on MuJoCo tasks. We follow the same parameter fixing as Snowflake. As in Snowflake, we parameterize F.,, and Fout as ML.Ps and the update function U GRU. We use weight matrix for M.
Pretrained NerveNet-Snowflaee Snowflake is a sfate-of-the-ee

---

Pretrained MetaMorph: MetaMorph [3] is Transformer based approach for learning universal controller over large collection of robot morphologies. We adopt MetaMorph's Transformer architecture for the policy network, which adds learned positional embeddings before processing the input sequence with a Transformer encoder. As our domains lack exteroceptive ohservations, we directly decode Transformer encodings to controller outputs. The Transformer policy is finetuned during RL.
5.2
Results
(MLP)
RL (Moduarr)
Pretrained NerveNie-Cony
Pretroined Nerveket- Srowt ake
Pretrained MetaMMrpph
MeMo
8 to 12 Leg Canlipade
to 10 Worm
to 10 Leg Hybrid
to 5 Finger Clew
2000
2000
3500
Reward
1540
1500
1000
500
Timesleps
Figure 6: Structure Transfer Results. Left: 6 leg centipede to 12 leg centipede transfer on the
Frozen Terrain. Left Middle: 6 leg worm to 10 leg worm transfer on the Frozen Terrain. Right Middle: 6 leg hybrid to 10 leg hybrid transfer on the Frozen Terrain. Right: 4 finger claw finger claw transfer on grasping cube. The dashed orange line shows that the final performance of the closest baseline is achieved by MeMo within half of the total number of timesteps.
Frozen Ridged Terrain
Gap Terrah
Frozen Stepped
Cube Sphere Grasping
2030
200
25000
12:9
X0NEC
1407
3900
Reward
1403
1309
1300
1600
330
000
300
284
080
205
445
660
062
360
265
400
560
Timesteps
Figure 7: Task Transfer Results. Left: The first three plots show results on transferring from the 6 leg centipede walking over the Frozen Terrain to the same centipede walking over terrain with ridges, a terrain with gaps, and a terrain with upward steps. Right: The last plot shows the transfer from a 4-finger claw grasping cube to the same claw grasping sphere. MeMo either has comparable training efficiency to the strongest baseline or outperforms all baselines.
The generalization ability of MeMo on structure transfer shown Fig. 6 On all structure transfer tasks, MeMo outperforms the message-passia passing baselines. On the 12 leg centipede and the 10 leg hybrid, not only is MeMo 2x more sample efficient than the best baseline, but also converges to controllers with significantly better performance than any baseline. On the 10 leg worm, MeMo outperforms all basclines in terms training efficiency and achieves a comparable final performance as NerveNet-Conv. McMo also outperforms all basclines the finger claw. We note that the worm transfer task is easier for GNN models, because the coordination of the shorter legs and body joints is naturally captured with multi-hop communication. MetaMorph struggles with locomotion tasks, due to the high dimensionalities of the transfer robots.
The results of MeMo on task transfer are shown in Fig. As transferring from the Frozen to the Ridged, Gap, and Stepped Terrains requires the robot to overcome obstacles unseen the Frozen Terrain, we load the pretrained boss controller and finetune MeMo end-to-end. Results (Fig. 7 show that on all test tasks, MeMo achieves improved training efficiency compared to MetaMorph and to pretrained MLP and modular architectures. McMo achieves comparable performance on the Ridged
7

---

and Gap Terrains and outperforms the NerveNet baselines on the Stepped Terrain, which requires the robot climb up steps whereas the training terrain is flat. MeMo also has improved training efficiency and final performance in the grasping domain when transferring from grasping cube to sphere. The pretrained NerveNets struggle to coordinate the arm and claw components, resulting in high variance across different random seeds.
5.3 Ablation Study
Sum of Modularity Objectives. We answer the question of why we choose to optimize the noise injection objective rather than the sum of the modularity objectives directly. We evaluate the sum of Eq. and 2 between networks trained with the noise injection objective and those trained with the sum in Table Using 100 sampled trajectories from the expert controller, average the resulting sum of objectives over 1000 epochs, with different sampled noise on each epoch. Our results demonstrate that optimizing the noise injection objective converges better solutions.
Table 1: Sum of objectives. On all starting morphologies, optimizing the noise injection objective results in lower loss than directly optimizing the dual loss.
MORPHOLOGY
NOISE INJECTION
DUAL Loss
CENTIPEDE
-33.518
-33.115
WORM
-39,295
-35,896
HYBRID
-30.536
-27.849
CLAW
-8.215
0.279
Noise Injection Objective. The key to the success of MeMo the introduced noise injection (NT) objective which encourages proper responsibility division among the pretrained boss controller and modules, enabling the modules improve training efficiency when reused. We conduct an ablation study to verify this technique by experimenting on a special setting, ""ransferring" the controller to the same robot structure and task, leg centipede traversing a Frozen Terrain. During transfer, we reuse and freeze the pretrained modules and retrain the boss controller from scratch. With the pretrained modules from MeMo, the boss controller will be retrained much more efficiently because it only needs to take partial responsibility for the control job. We compare our method to three baselines;
MeMo (no NI): We pretrain the modular architecture end-to-end without noise injection. This ablation is equivalent to MeMo without noise injection.
MeMo (L1): During pretraining, we replace the injected noise with L1 regularization on B's output that encourages sparsity in its signal. We weigh the regularization term by hyperparameter w and report results with the best w.
MeMo (I22): During pretraining, we replace the injected noise with 1.2 regularization on B's output and report results with the best weight on the regularization term.
MeMo (Jacobian): As alternative to noise injection, we penalize the norm of the module's Jacobian using the method described 19].
In addition, we add the training curve of RL (Modular) as a reference. The results (Fig. 8 show that MeMo yields significant improvement in training efficiency over all ablations.
Ablations
Ablations
2300
2000
(Modular)
3390
McMo (L.1)
NII
1520
Pretrained kitl Pretrained wi SNI Meloo
(Mudulan)
Meloo (12)
MoMo (acobiaan)
Timeeleps
MeMo outperforms all other variants that are pretrained with II. Right: MeMo outperforms all variants that pretrain modules with RL. In both settings, MeMo achieves the final performance of the closest baseline within half of the total number of timesteps.
Figure 8: Ablation Results. Left:

---

Imitation Learning. We now answer the second question of whether imitation learning is necessary to pretrain modules with Gaussian noise injection. The results of using noise injection reinforcement learning to pretrain modules is shown in Fig- g Note that we refer to II. ablations as experiments where modules are first pretrained with imitation learning, and subsequently, the boss controller is reinitialized and retrained with RL to test the improvement in sample efficiency. RL ablations involve the second RL phase, but the pretraining stage is done with RL as well. In addition to training the modular architecture from scratch, we experiment with two methods of injecting noise during RL. The first is naive noise injection (NNI), where we inject noise into B's output when sampling rollouts and computing policy gradients. For the second, we adopt the Selective Noise Injection (SNI) technique proposed by [20] for applying regularization methods with stochasticity in RL. SNI stabilizes training by sampling rollouts deterministically and computing the policy gradient mixture of gradients from the deterministic and stochastic policies. However, even with SNI, the pretrained modules do not improve training efficiency.
5.4 Analysis
We examine how the noise injection objective forces the modules to learn better representation of the actuator space. As discussed in Section [ the trajectories produced by a successful policy often lie on much lower-dimensional manifold than the actuator space. Each dimension of the manifold can be interpreted as an individual skill that the policy has learned. We can measure the dimensionality of the modules' mapping by looking the Jacobian matrix of the worker modules with respect to the boss's signal. The trajectories outputted by a policy can likely be captured by a few dimensions of high variance corresponding much larger set of dimensions of lower variance corresponding to relatively small singular values.
small set of large singular values in addition to
We visualize this effect by 1) computing the Jacobians at the trajectory input states of a suc- cessful policy and 2) normalizing the singular values of each Jacobian by its largest singular value and plotting the resulting values in the [0,
MeMo MeMo (No NI)
60000
50000
I] range. We expect that a module that optimizes the invariance to noise objective will have only small number of large singular values, with the
40000
Count
30000
rest being close zero. Conversely, modules that do not produce a low-dimensional mani- fold would have more singular values of similar magnitude, resulting in the distribution's 's mass clustering close 1. We verify this intuition by sampling 100 trajectories from an expert con- troller for the 6 leg centipede shown Fig. [ We compare the plots of the normalized singular values between MeMo and MeMo without noise injection in Fig. 9 Without noise injection, the majority of the values are close to 1. At the other extreme, with MeMo, the values are highly clus- tered to the left, implying that most singular val- ues much smaller than the biggest singular value. We plot the singular value distributions of additional MeMo ablations in Appendix A.
20000
10000
8.0
0.2
0.4
0.6
0.8
1.0
Normalized Singular Values
Figure 9: Singular Value Distributions of Actuator-Boss Jacobians. For modular architec- tures trained with and without the noise injection, we plot the normalized singular values of Jaco- bian matrices over an expert's 'sss trajectories. With noise injection, the mass of the distribution much closer to O, showing that the modules learn better representations of the actuator space.
Conclusion
In this paper, we propose modular architecture for robot controllers, in which a higher-level boss controller coordinates lower-level modules that control shared physical assemblies. We train the architecture end-to-end with noise injection, which ensures that the lower-level modules do not overrely on the boss controller's 's signal. In locomotion and grasping environments, we demonstrate that our pretrained modules outperform both GNN and Transformer-basedd methods when transferring from simple to complex morphologies and to different tasks. We ablate components of MeMo and demonstrate that the entire framework is necessary to achieve these generalization benefits.

---

Acknowledgments and Disclosure of Funding
MT is supported by the National Science Foundation (NSF) under Grant No. 2141064. AS is supported by the National Science Foundation (NSF) under Grant No. 1918839 and by the MIT-IBM Watson AI Lab. MT was also additionally supported by the MIT Stata Presidential Fellowship. This work greatly benefited from discussion with colleagues in the MIT Computational Design and Fabrication Group and MIT Computer- Aided Programming Group. Any opinion, findings, and conclusions or recommendations expressed in this material are those of the authors(s) and do not necessarily reflect the views of the funding entities.
References
[1] W. Huang, I. Mordatch, and D. Pathak, "One policy to control them all: Shared modular policies for agent-agnostic control," in Proceedings of the 37th International Conference Machine Learning. 2020.
[2] V. Kurin, M. Igl. T. Rocktäschel, W. Boehmer, and S. Whiteson, "My Body is a Cage: The Role of Morphology in Graph-Basee Incompatible Control," in International Conference on Learning Representations, 2021.
[3] A. Gupta, I.. Fan, S. Ganguli, and L.. Fei-Fei, "MetaMorph: Learning Universal Controllers
with Transformers," in International Conference on Learning Representations, 2022.
[4] T. Wang, R. Liao, J. Ba, and S. Fidler, "NerveNet: Learning Structured Policy with Graph Neural Networks," in International Conference on Learning Representations, 2018.
[5] C. Blake, V. Kurin, M. Igl, and S. Whiteson, "Snowflake: Scaling GNNs to high-dimensional continuous control via parameter freezing," in Advances in Neural Information Processing Systems, vol. 34, 2021.
[6] Chen, J. Xu, and P. Agrawal, "A System for General In-Hand Object Re-Orientation," in 5th Annual Conference
Robot Learning, 2021.
[7] H. Zhu, A. Gupta, A. Rajeswaran, S. Levine, and V. Kumar, "Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Cost," in 2019 International Conference on Robotics and Automation (ICRA), 2019.
[8] Radosavovic, X. Wang, L. Pinto, and J. Malik, "State-Only Imitation Learning for Dexterous Manipulation," in International Conference on Intelligent Robots and Systems, IROS 2021, 2021.
[9] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, "Proximal Policy Optimization Algorithms," CORR, vol. abs/ 0774777777 2017.
[10] S. Ross, G. Gordon, and D. Bagnell, "A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning." in Proceedings of the Fourleenth International Conference on Artificial Intelligence and Statistics, 2011.
[11] C. Devin, A. Gupta, T. Darrell, P. Abbeel, and S. Levine, "Learning modular neural network policies for multi-task and multi-robot transfer;" in 2017 IEEE International Conference on Robotics and Automation (ICRA), 2017.
[12] J. Whitman, M. J. Travers, and H. Choset, "Learning Modular Robot Control Policies," CORR, 2021.
vol. abs/2105.
[13] D. Pathak, C. Lu, T. Darrell, P. Isola, and A. A. Efros, "Learning Control Self-Assembling Morphologies: A Study of Generalization via Modularity," in Advances in Neural Information Processing Systems, vol. 32, 2019.
[14] D. Hejna, L. Pinto, and P. Abbeel, "Hierarchically Decoupled Imitation For Morphological Transfer," in Proceedings of the 37th International Conference on Machine Learning, 2020.
[15] X. Liu, D. Pathak, and K. Kitani, "REvolveR: Continuous Evolutionary Models for Robot- to-robot Policy Transfer," in Proceedings of the 39th International Conference on Machine Learning, 2022.
[16] A. Zhao, J. Xu, M. Konaković-Luković, J. Hughes, A. Spielberg, D. Rus, and W. Matusik, "RoboGrammar: Graph grammar for terrain-optimized robot design," ACM Transactions on Graphics, vol. 39, no. 6, 2020.
10

---

[17] J. Xu, T. Chen, L. Zlokapa, M. Foshey, W. Matusik, S. Sueda, and P. Agrawal, "An End-to- End Differentiable Framework for Cwwte Robot Design," in Proceedings of Roboties: Science and Systems, 2021.
[18] T. N. Kipf and M. Welling, "Semi-Supervised Classification with Graph Convolutional Net- works," in International Conference on Learning Representations, 2017.
[19] J. Hoffman, D. A. Roberts, and S. Yaida, "Robust Learning with Jacobian Regularization." 2019.
[20] M. Igl. K. Ciosek, Li, S. Tschiatschek, C. Zhang, S. Devlin, and K. Hofmann, "Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneck," " in Advances in Neural Information Processing Systems, vol. 32, 2019.
[21] M. Bowers, T. X. Olausson, L. Wong, G. Grand, J. B. Tenenbaum, K. Ellis, and A. Solar- Lezama, "Top-Down Synthesis for Library Learning." " Proc. ACM Program. Lang., vol. 7, no. POPI., 2023.
[22] D. Cao, R. Kunkel, C. Nandi, M. Willsey, Z. Tatlock, and N. Polikarpova, "Babble: Learning Better Abstractions with E-Graphs and Anti-unification." Proc. ACM Program, Lang., vol.
no. POPI., 2023.
[23] R. S. Sutton, D. McAllester, S. Singh, and Mansour, "Policy Gradient Methods for Reinforce- ment Learning with Function Approximation," in Advances in Neural Information Processing Systems, vol. 12, 1999.
[24] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and Moritz, "Trust Region Policy Optimization," in Proceedings of the 32nd International Conference on Machine Learning, 2015.
[25] A. Camuto, M. Willetts, U. Simsekli, S. J. Roberts, and C. C. Holmes, "Explicit Regularisation in Gaussian Noise Injections," in Advances in Neurul Information Processing Systems, vol. 33, 2020.
[26] O. Dhifallah and Y. Lu, "On the Inherent Regularization Effects of Noise Injection During Training," " in Proceedings of the 38th International Conference on Machine Learning, 2021.
[27] S. H. Lim, N. B. Erichson, L. Hodgkinson, and M. Mahoney, "Noisy Recurrent Neural Networks," in Advances in Neural Information Processing Systems, vol. 34, 2021.
[28] C. Florensa, Y. Duan, and P. Abbeel, "Stochastic Neural Networks for Hierarchical Reinforce- ment Learning," in International Conference on Learning Representations, 2017.
[29] Y. Li, Y. Wu, H. Xu, X. Wang, and Y. Wu, "Solving Compositional Reinforcement Learning Problems via Task Reduction," " International Conference on Learning Representations, 2022.
[30] S. Kumar, J. Zamora, N. Hansen, R. Jangir, and X. Wang, "Graph Inverse Reinforcement Learning from Diverse Videos," in Conference on Robot Learning, CoRL 2022, K. Liu, D. Kulic, and J. Ichnowski, Eds., vol. 205, 2022.
[31] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, D. Ho, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. J. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, S. Levine, Lu, L. Luu, C. Parada, Pastor, J. Quiambao, K. Rao. J. Rettinghouse, D. Reyes, P. Sermanet, N. Sievers, C. Tan, A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, S. Xu, and M. Yan. "Do As I Can, Not As I Say: Grounding Language in Robotic Affordances," " CoRR, vol. abs/2204.01691, 2022.
[32] W. Huang, P. Abbeel, D. Pathak, and I. Mordatch, "Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents," in International Conference on Machine Learning, ICML 2022, K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, Eds., vol. 162, 2022.
[33] H. Furuta, Y. Iwasawa, Y. Matsuo, and S. S. Gu, "A System for Morphology Task Generalization via Unified Representation and Behavior Distillation," CoRR, vol. abs/2211. 1..22.1..444 14299,, 2022.
[34] Y. Lee, Yang, and J. J. Lim, "Learning Coordinate Manipulation Skills via Skill Behavior Diversification, in International Conference on Learning Representations, 2020.
[35] E. Aljalbout, M. Karl, and P. van der Smagt, "CLAS: Coordinating Multi-Robot Manipulation with Central Latent Action Spaces," in Proceedings of The 5th Annual Learning for Dynamics and Control Conference, 2023.

---

A Appendix
A.1 Limitations and Future Work
We now discuss the limitations of our work and potential future directions. One limitation is that our experiments were conducted using only single type of RI. algorithm (PPO) and network architecture, MLPs with Tanh nonlinearities and orthogonal initialization. Future work would involve applying and adapting our framework to different policy optimization algorithms and architectures.
Because our approach involves imitation learning to pretrain the modules, we acknowledge that there is a potentially greater memory footprint for storing the imitation learning dataset.
While we have demonstrated the potential of MeMo to be used for task transfer, such capabilities are inherently limited as our architecture does not explicitly encode the semantics of the task. For example, transferring the modules of a 6 leg robot walking over Frozen Terrain to the same robot traversing a terrain with obstacles requires finetuning the architecture end-to-end, which may not always yield better results due to the instability of policy updates. Combining our framework with a mechanism for representing task semantics to enable transfer to more complex tasks is promising direction for future work. In addition, as our experiments are only in simulation, an important line of future work is applying our approach to real world tasks.
In general, we would expect our framework to face the same challenges in adapting to the real world as standard RL policies, as our problem of learning modules that generalize to different morphologies orthogonal to learning policies that overcome the sim-to-real gap. Extending our framework to deal with challenges in sim-to-real transfer. including adapting to environmental variations, hardware inconsistencies, and discrepancies in simulated physics vs real dynamics, is an important avenue for future work.
The scope of our work limited robots that are incrementally different from the starting robot, due to the difficulty of generalizing from a single robot and environment. One line of future work could involve adapting our training pipeline multirobot and multitask settings, enabling our modules to capture broader range of robot dynamics. When the dataset of robots grows larger, it can be expensive for a domain expert to manually provide labels on how a robot is decomposed into physical assemblies. However, the problem of learning reusable components bears similarity to the problem of abstraction learning studied in the programming languages community. Recent advances [2T 22 have made abstraction learning much more computationally efficient than in the past. We also see promise in adapting these techniques, which have been developed for programs, to robots that have an underlying graph structure.
A.2 Broader Impact
Here discuss the broader social impact of our work, including its potential positive and negative aspects. On the positive side, our work enables us to train neural network architectures which are structured in a more interpretable manner, in that the modules correspond to physical components of the robot. In addition, we demonstrate generalization to robot structures and tasks that are greater in diffieulty than the training setting. In summary, our modular approach is a step towards addressing the concern that neural networks are black-box models with highly limited generalization capabilities.
We do not any direct negative implications stemming from work, as experiments are solely conducted in simulated robot environments. We note that our work does not impose safety constraints on the rollouts of the agent, which is important limitation to address for real-world use of our method.
A.3 Further Architecture Details
As shown in Fig. I0, the modular architecture starts by executing the boss controller B, which takes the full observation vector s" as input. The full observation vector s" includes both global observations about the agent and local observations of each actuator. The global observations consist of the agent's global position, orientation, and velocity, while the local observations are composed of joint angle, joint velocity, and local relative position and orientation in the corresponding module's frame. Given the full observation vector s", M outputs a latent vector H of length [P]- D2 where D is the size of the embedding sent to each module.
12

---

Boss
Nosse
Shared Modules
W;
W;
Agant's glcbal posttion, velocity, orientetion
Ju position, velocity, orientation
+ +1100, e)
W3
JN-1 position, velocity orientation
Wi
Figure 10: Modular Architecture with Noise Injection. Our architecture consists of higher-level boss controller that outputs hidden embedding, æ. During imitation learning, Gaussian noise is added to æ to compute H. H is split into signals that are passed into modules that output the mean of the action distribution. The dotted lines represent that addition H, the modules also take in subsets of the full observation vector corresponding to the state of the joints within the modules.
The latent vector H is then split into 'Pl segments of size D) and fed to modules. As shown in Fig. module itself consists of MLP for each actuator Jn. Each MLP takes input i. 's local features concatenated with the module's segment of the latent vector H and outputs the mean value of the action applied to jn.
(w)()")
f
W
Ja MLP
M.P
O
65
Hw?
(a) W Subgraph
(b) W Architecture
Figure 11: Module Subgraph and Architecture. Left: Module w is responsible for computing the mean actions of actuators and 1. Right: A module consists of separate networks that compute each actuator's mean action. The inputs include the local observations of the actuator concatenated with the signal sent to the module belongs to.
A.4 Decomposition of Noise Injection Loss
Our goal is show that Eq. 3.1 equivalent to the sum of Eq. and with remaining product term. We can decompose the noise injection loss as follows:
= E,, Ex (B. + F(ss))
(6)
E., E. (D(s,, + (W, F(s:)))
(7)
- [E,,LE,,((s,,, (W,
- F(s:)
+ (Wo =(
(8)
-E (Wo - F + B E, D(s,, 1
+ E. E, (W,
(s,) (B,, -
F(s)))]
(9)
13

---

A.5 Negative Log Likelihood Loss
In practice, we minimize the negative log likelihood loss during imitation learning. Similar the above derivation, show that negative log likelihood with noise injection can be written in terms of the behavior cloning and invariance to noise losses. Let our modular policy, where the mean of the Gaussian distribution for each of the actions is given by W (Bo((()))) and the standard deviation is trainable vector, du. We define the behavior cloning and invariance noise losses as negative log
likelihoods in Eq. 11 and 13 respectively.
Ps (F(s)) 184) = 1
exp
(W (Ba(s)))) - F(s:))²
(10)
Ja 2
20
- E, Pr Pr (F(s:) 8:)] = Es log( a 22T) +
W (Bo(s:)) - F(8:))
(11)
20
1
Ps (Wa(Ba())) 84,7) ou ou
exp
20
(W (Bo(B()))) + - (Be(()))))
(12)
-Bn [E [log Px (Wo(Bo(s))) (ss.))) n)]l = B, Es log(o, u v 2 + 20
(W (Bo +
W (B
(13)
We now consider the conditional log likelihood with noise injection and show how it can be decom- posed in terms of the above losses.
Pa (F(ss) St 1)
1 exp
1
(Wo (Bo(s) + +1) F(s.))²
(14)
o, 2m nx
20
E, [E, [log pM (F(s:)
ss. E,
EN log(o V2T)
(W (Ba(s) + +1) F(s.))2
20
(15)
Ez
log(on (2T) (W (Wo(Ba(s:)) F(s:))²
20
+ B.
(Wa (B)(())) + Wa
20
(16)
=-E, [log P. (F(s:) s:)] - E., [E [lo P.. (Bo(s)) 8; m)]
-log(o, 22x C
(17)
where C' is the product term.
A.6 Further Experimental Details
Let D be the base hidden size of the network. As typical in PPO, we use Tanh nonlinearities and orthogonal initialization for the standard MLP and modular architectures. The standard MLP and boss controller are layer neural networks. The size of the first layer is D while the second layer has L D hidden units, where L is the number of modules. The standard MLP also has final linear layer to decode the actions. For all policy architecture variants, the value function is defined as a layer neural network with D hidden units each, followed by a linear layer.
In PPO, agents iteratively sample trajectories based on current policy and subsequently perform optimization on surrogate objective that first-order approximates the natural gradient. The surrogate objective prevents unstable updates to the policy by clipping the probability ratio r" (0; 0ata) To(a" s² s²)) TT (a" s Optimizing the clipped objective is done with the policy gradient [2]
14

---

The RL loss that all architectures optimize includes the surrogate objective, weighted value function loss, and weighted entropy bonus encourage exploration:
L L² E (L (,) c25
E min (9), (0), c, [))
c, (Vo())) - Vary (s )) + c2 [5(m0(())
(18)
where A is the generalized advantage estimation (GAE) 24 is the clip value, is the weight on the value function, and 02 used to balance the entropy bonus.
We use Adam as the optimizer for both RL and IL. Semicolon-separated entries denote different values for the two domains: "[Locomotion Value]; [Grasping Value]". We conduct extensive hyperparameter search and find that that the values in Table yield reasonable performance.
Parameters
Value Set
Value Loss Factor CI Entropy Bonus Factor
0.5
0
Discount Factor y
0.995
GAE X
0.95
PPO Clip Value
0.2
Gradient Clip Value
0.5
Starting Learning Rate
3e-4
Number of Iterations per Update
10
Learning Rate Scheduler
Decay
Number of Processes
8; 16
Batch Size
2048; 100
Number of Timesteps
8e6; 3e5
Base Hidden Size D
128; 64
Table 2: RL Hyperparameters
For experiments with pretrained policy weights, we initialize the learned logstd to -1.0. For all models on the locomotion tasks, we perform a secondary search over the batch size [256, 512, 1024, 2048] until performance decays. We find that MeMo works best with smaller batch sizes: 256 on the 12 leg centipede, 512 on the 10 leg worm, and 1024 on the leg hybrid. The modular architecture also sees improvement when using 1024 for all three robots, whereas the default batch size of 2048 works best for the MLP architecture. NerveNet- size of 1024 on the 12 leg centipede and 10 leg worm. We do not see significant improvement when decreasing the batch size for MetaMorph. Due to the higher variance in reward for grasping tasks, we keep the same batch size for the transfer experiments.
and NerveNet- Seaveeee--nnofllade Snowflake improve with a batch
For the modular architectures, we parameterize each joint network within the modules with a layer MLP with 32 hidden units per layer. For imitation learning, we use batch size of 1024 and tune the learning rate in [7e-4, le-3, 2e-3, 4e-3, 7e-3). All models are trained for 175 iterations of DAgger. We sample 500 trajectories from the expert controllers of the 6 leg centipede, 6 leg worm, and 6 leg hybrid and 250 trajectories from the controller of the finger claw as the validation sets. We make sure that the architectures pretrained with imitation learning achieve comparable average reward as the expert controller when a number of trajectories are sampled from them.
For NerveNet, the input network is single layer with size D followed by a Tanh nonlinearity. We have a separate output network for each joint, and each output network a 2 layer MLP with 32 units layer. Table summarizes the hyperparameter search that we perform for NerveNet. We perform grid search over the number of layers and the size of the messages passed by the propagation network. We choose the smallest architecture size that achieves a similar average reward as MeMo. Adding skip connection from the root to all joints improves NerveNet's validation score in imitation learning and enables the use of smaller architectures that do not overfit easily.
15

---

Parameters
Value Tried
Number of Layers
2, 3, 4 32, 64, 128
Message Size
Skip Connection
Yes, No
Table 3: NerveNet Hyperarameter Search
For MetaMorph, we tune the number of attention layers in [2, 3, 4] and otherwise use the same architecture hyperparameters as [3]. listed in Table For a fair comparison, we use MLP critic with the Transformer policy during RI. rather than a Transformer critic, as the critic trained from scratch during transfer. We find that positional encoding improves imitation learning pretraining, enabling the use of smaller architectures. For each token that corresponds to joint, we include the one-hot encoding of the joint type in the observations.
Parameters
Value
Number of Attention Heads Embedding Dimension
128
Feedforward Dimension
1024
Nonlinearity Dropout
ReLU
0.1
Table 4: Transformer Hyperparameters
A.7 State Space Description
We keep running mean and variance to normalize the state space. Relative positions / orientations are relative to a joint in the same module given joint. For grasping, we use relative joint orientations as global joint orientations depend significantly on how high the claw is lifted. Table 5 and 6 detail the observation space in locomotion and grasping. For locomotion, "base" refers the forwardmost wide body segment of the robot. As the joints are hinge joints, they only have one degree of freedom. The token type refers the observation processing for MetaMorph each input sequence consists of single "global" token with the corresponding global observations for the robot concatenated with zero padding for the local observations, and the rest the tokens are "joint" tokens with zero padding for the global observations concatenated with the local observations of the corresponding joint. We choose to integrate global information at the encoder level rather than the decoder level as our global features are low-dimensional: only 16 dimensions at most. The original MetaMorph architecture considers exteroceptive features from camera or depth sensors as global, which are much higher dimensional and are concatenated at the decoder level to prevent the dilution of local proprioceptive information.
A.8 Computing Infrastructure
We run experiments 2 different machines with AMD Ryzen Threadripper PRO 3995WX processors and NVIDIA RTX A6000 GPUs Both machines have 64 CPU cores and 128 threads. The main cost of running the agent in both the RoboGrammar and the DifiRedMax environments is the cost of simulation, which is CPU-intensive. For the MLP-based architectures, we only use CPU cores for computing rollouts in parallel environments via vectorization and backpropagating the policy gradient. For NerveNet, in the locomotion domain, we find it helpful to vectorize environments while performing backpropagation with GPU. For example, the RL stage of MeMo on the 6 leg centipede takes less than day to complete, whereas training 3 layer NerveNet-Com with the same number of processes and batch size requires 3-4 days without GPU. We note that our resources are shared, and the wallclock time varies depending on the other processes running on the same server.
A.9 Additional environment details
We provide more details on the RoboGrammar tasks. all locomotion tasks, the maximum episode length is 128. Full details of the environments can be found in the RoboGrammar codebase [16).
16

---

Table Locomotion Observation Space
Controller Type
Node Type
Token Type
Observation Type
Axis
base position
base velocity
base velocity
base velocity
boss
root
global
base angular velocity
X
base angular velocity
base angular velocity
base orientation
base orientation
base orientation
joint position
joint velocity
joint orientation
boss, module
joint
joint
joint orientation
joint orientation
joint relative position
X
joint relative position
Y
joint relative position
Z
Table 6: Grasping Observation Space
Controller Type
Node Type
Token Type
Observation Type
Axis
relative fingertip position object
boss
root
global
relative fingertip position to object
y
relative fingertip position object
joint position
joint velocity
joint relative orientation
X
boss, module
joint
joint
joint relative orientation
y
joint relative orientation
Z
joint relative position
X
joint relative position
y
joint relative position
7.
Frozen Terrain: A flat surface with friction coefficient of 0.05.
Ridged Terrain: Ridges are placed average of one meter apart across the width of the terrain.
Gap Terrain:
A series of platforms separated by gaps.
Stepped Terrain: A series of steps with varying height, resembling a flight of stairs.
For all RoboGrammar locomotion environments, the reward at timestep is the sum the rewards at each sub-step t. The training reward function at substep t is
R(st, at) = Vg + (e body
world
body
Cyy
world) - |.
(19)
where N is the dimension of the action vector, and each dimension is normalized to [-1, 1]. The first two terms encourage high velocity in the z-direction and maintaining the robot's initial orientation respectively. The last term is regularization penalty to reduce the variance across different runs. The reported reward curves do not include the regularization penalty.
For grasping, the goal is to grasp an object and lift as high possible and the maximum episode length is 50. As prior work [6], we follow the convention of controlling the actuators with relative positions rather than absolute positions. The reward at timestep is the sum of the rewards at each sub-step l. The full set parameters used construct the DiffRedMax simulation will be released with our source code. Below is the reward function used, where object, refers to the object's 7-coordinate and avg__ingg www.gfnniitppccc dist the mean distance of the claw's fingertips the object's surface. We approximate the cube's surface with the surface of the largest sphere that fits in the cube.
17

---

all fingers in contact checks whether or not all fingers of the claw is within a small distance from the surface of the object.
[10 object_ 0.1 awg_fingertip dist all fingers in__ontact !all dill_fingrr_ine in_contact
R(st, as) = -- avg fingertirttp
-
The penalty avg__ingeeeaaaeeaaae fingertip_ dist encourages the fingers to grasp the object. We only include the reward term on object, when all fingers__inn pesers_ii__oon in_contacttttttttoeoeeeoeeeee satisfied order to prevent the claw from throwing the object.
A.10
Sources
We use the PPO implementation provided in http:///hub..ccoooottrro..o pytorch- aath-aaccppPa- ppo-pppoaaaaeeeee acktr- gaail| (MIT License). Our NerveNet implementation adapted com/ ctape/o.ooo/aaaaeaeee Homm/HnnnesStakk/ gun-reinfor -larninng We adapt our MetaMorph implementation from the official codebase: https: https://giae //ggttttt.../...//gg
from a PyTorch version of the original NerveNet codebase: https://githhi
We use the official (MIT License) simulators.
RoboGrammar 16
(MIT License) and DiffRedMax
A.11 Extended Related Works
Noise Injection. One line work focuses on explaining the generalization benefits induced by noise injection by deriving explicit regularization terms. [25] studies Gaussian noise injected into network activations each layer and derive an explicit regularization term by marginalizing out the noise. (26) analyzes the effect of Gaussian noise injection to the training data and find that the effect is equal to weighted ridge regularization as the number noise injections approaches infinity. [27] study injecting noise into RNN hidden states and identify an explicit regularizer for small noise variances.
Hierarchical and Multi-Task RL. Our proposed modular architecture bears similarity to those used in hierarchical RL 228 14]. However, key difference is that our architecture is hierarchical with respect to the morphology of the robot, not the temporal structure of the task. To train robots that perform a diverse set of skills and generalize to new tasks, prior work leverages the shared structure of tasks, such as through graph representations [29 30 that represent task compositionality, or through
language representations 31 32). While many works in MTRL focus on single morphology. recent efforts 3K have proposed representing both morphology and task in single graph, enabling architectures trained on this unified IO representation to transfer to unseen morphologies and tasks.
Multi-Robot Coordination. Past works in multi-robot coordination bear similarity to our work in either the modularity of the architecture or the learning of higher-level coordination mechanism, analogous boss controller, between different agents. In particular [4] uses a modular architec- ture, in which a higher-level meta-policy coordinates various skills with behavior embedding. (35) proposes learn useful latent action space for coordinating a multiagent system via an information bottleneck. The information bottleneck helps in learning a latent action space from the full set of observations that is useful in coordinating decentralized agents at inference time.
A.12 Additional Experiments
In Fig. 12 we test the zero-shot generalization of the pretrained NerveNet-Com baseline by fixing all of its weights and only training the learned standard deviation when transferring from the to the 12 leg centipede. Its poor performance demonstrates the difficulty of the transfer task, in spite the physical similarities between the 6 and the 12 leg centipede.
In addition, we demonstrate the capability of MeMo is scaling to much higher dimensionalities by transferring from a 6 to a 24 leg centipede (Fig. 13) on Frozen Terrain. MeMo significantly outperforms RL (Modular), the strongest baseline the 6 to 12 leg centipede transfer task.
In Fig. we show that our framework has the potential to correct the dynamics of the system during transfer, specifically when some joints in the transfer robot fail to perform as expected even when given the correct control signal. In the 6 to leg centipede transfer, we randomly select out of 70 joints in the 12 leg centipede to be uncontrollable (for each seed of the experiment, different subset
18

---

Pretrained NerveNet-Com Fixed all
470
460
450
440
430
2e8
4e5
8e6
8e6
Timesteps
Figure 12: Fixed NerveNet-Conv on 12 Leg Centipede Transfer. All weights of the NerveNet- Conv baseline, pretrained on the leg centipede, are fixed during transfer to the 12 leg centipede on Frozen Terrain, resulting in suboptimal performance.
RL (Modular)
MeMo
1400
1200
1000
Reward
600
400
200
0
066
2e6
6e6
Timesteps
Figure 13: 6 Leg to 24 Leg Centipede Transfer Results. The dashed orange line shows that the final performance of the closest baseline is achieved by MeMo less than half of the total number of timesteps.
of uncontrollable joints is sampled, and we have 3 random seeds). For the uncontrollable joints, we pass in a small random noise instead of the controller's output the simulator. MeMo significantly outperforms the RL (Modular) baseline, achieving its final reward in less than half of the timesteps.
In Fig. [5] test the transfer capabilities of MeMo morphologies smaller than the starting structure. Specifically, we transfer modules from a 6 leg to a leg centipede on the Frozen Terrain and from 4 finger to 3 finger claw grasping cube. MeMo achieves improved training efficiency to policies trained from scratch and performs similarly to the strongest baseline each domain.
In Fig. 16 compare the singular value distributions of MeMo to the ablations described in Section 5.3land and MeMo with a lower value of a. In Fig. 17 we run MeMo on the 6 to 12 leg centipede transfer where either the boss controller or the modules are layer MLPs instead of 2. Both of these variants perform similarly to the original architecture.
19

---

RL (Modular)
MeMo
1750
1250
Reward
1000
750
500
250
0
0e6
2e6
426
6e6
Timesteps
Figure 14: 6 Leg to Broken 12 Leg Centipede Transfer Results. MeMo achieves the final performance of the closest baseline in less than half of the total number of timesteps.
Herrwhel-CCarv
Pretzained Sarvwelei-Snoowiiee
Pretzained
Centipade
Finger Claw
25000
2003
22000
Reward
15000
10000
565
Timestaps
Figure 15: Complex to Small Structure Transfer Results. Left: leg centipede to 4 leg centipede transfer on the Frozen Terrain. Right: 4 finger claw to finger claw transfer on grasping a cube. On both transfer tasks, MeMo achieves comparable performance to the performance of the strongest baseline.
70000
MeMo
MeMo
50000
60000
MeMo (L1)
MeMo
MeMo (Jacobian)
(Noise=0.5)
50000
40000
Count
40000
Count
30000
30000
20000
20000
10000
10000
8.0
8.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6 0.8
1.0
Normalized Singular Values
Normalized Singular Values
(a) McMo Ablations
(b) McMo with Less Noise
Figure 16: Additional Singular Value Distributions. Left: For various ablations of MeMo, we plot the normalized singular values of Jacobian matrices computed over an expert's trajectories. With noise injection, the mass of the distribution is much closer to O. Right: With injected noise sampled from a Gaussian distribution with standard deviation 0.5 instead 1.0, the mass of the distribution is closer to 1.
20

---

MeMo, 4 Layer Boss
MeMo, Layer Modules
MeMo
2250
2000
1750
1000
750
500
066
2o5
466
6eb
806
Timesteps
Figure 17: Architecture Variants of MeMo on 6 to 12 Leg Centipede Transfer: We run experi- ments where either the size of the boss controller the size of the modules is increased from 2 to layers. Both of these variants achieve comparable performance to the original architecture with layer MIPs.
A.13 Additional Figures
Cube
Frozen
Spbere
Ridged
Gap
Stapped
The training task is leg centipede locomoting the Frozen Terrain. The goal is to transfer policy weights to Ridged, Gap, and Stepped Terrains, all of which require the robot overcome obstacles unseen in the Frozen Terrain. Right: In the grasping domain, the training task is a 4 finger claw lifting cube, and the testing task is the same claw lifting sphere. sphere naturally harder object to grasp due its curved surface.
Figure 18: Task transfer. Left:
21

---

W
17
15
14
16
15
14
16
10
10
12
W
12
root
Figure 19: Graph Structure and Modules of the 6 Leg Worm. Left: Rendered robot, with joints labeled numerically and circled. Right: Corresponding graph structure with joints nodes and links as edges. The joints circled red can thought of the "head" while the joints circled in blue form the "body" modules.
w
W3
29
27
25
24 26
28
23
19 13
18
22
10
W!
15
14
12
8
w
5
2
W
3
foot
Figure 20: Graph Structure and Modules the Leg Hybrid. Left: Rendered robot, with joints labeled numerically and circled. Right: Corresponding graph structure with joints nodes and links as edges. The joints circled in red belong to the "leg" modules, those circled green belong to "body" modules, and those circled in blue belong the "head" module.
22

---

root
wg
0
W
w?
w
Figure 21: Graph Structure and Modules of the 4 Finger Claw. Left: Rendered robot, with joints denoted by red spheres. Right: Corresponding graph structure with joints as nodes and links as edges. Each pair of finger joints belongs in its own module, and the arm joint belongs separate module.
23

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect paper's contributions and scope?
Answer: [Yes]
Justification: Our claims on the positive transfer benefits of McMo supported by structure and transfer experiments in (.T] Our ablation experiments in Section 5.3 demonstrate that noise injection is the key component enable these benefits.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims
made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Appendix A.1
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms
and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]

---

Justification: See Appendix A.4 A.4for for a complete proof of the decomposition of noise injection into the sum of the two modularity objectives.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented
by formal proofs provided in appendix or supplemental material.
Thcorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: In Appendix A.33 we provide full details of our network architecture, and in A.6 we provide hyperparameters of RL and IL training.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [Yes]
Justification:
We have provided a link to our code in the main text.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
Bubllcc/guide/Codeeeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips wwtpcccppppccccccecccc.ccccccllt
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why. At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Section 5.1 for details on our experimental setup and Appendix A.6 for
details on hyperparameters.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them. The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: As stated in Section 5. we conduct compute statistics over runs for locomotion and 5 runs for grasping experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
26

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix A.8
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We have reviewed the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Appendix A.2
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal
impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
27

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: See Appendix A.10
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URI..
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pwwesswiteeeatteeeeeeeeeeee thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Besides code, we do not release new assets.
Guidelines:
28

---

The answer NA means that the paper does not release new assets. Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve research with croudsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve research with croudsourcing or human subjects,
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as institution conducting the review.
29


Reviews:
[{"review_id": "dGNh9O3RV1", "reviewer": "Reviewer_Hbiz", "summary": "This paper presents a method for learning modular controller that can be transferred and adapted to different morphology of robots and tasks. The high level idea is to decompose control of each motor through distilling a learned hierarchical RL policy and then uses these primitive policies as building blocks to additional motors added.", "questions": "Will this method generalize to multi-agent RL setting where the modular policy might be more complex than controlling 1-DoF motor?", "limitations": "see weakness", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper proposes a way to transfer low-level control policies to similar but different morphology of robots with a learned structure. The problem is novel and interestingly challenging. The proposed method is conceptually intuitive. Experiments in simulation demonstrate that the proposed algorithm can enable policy transfer across different robots and tasks.", "weaknesses": "The proposed method only trains the master policy at adaptation stage, which assumes the change of morphology does not induce change in dynamics of the system such that drastically different low-level policies will be needed to solve the task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the review and are glad that they find our problem novel and our proposed framework intuitive. Below we address the reviewer’s concerns and questions.\n\n**Under \"Weaknesses\":** \n\n> The proposed method only trains the master policy at adaptation stage, which assumes the change of morphology does not induce change in dynamics of the system such that drastically different low-level policies will be needed to solve the task.\n\nThis is correct, and we will be happy to make this clearer in the final version of the paper; we expect our method to be most effective in situations like the ones in our experiments, where the new morphology is performing a similar task to the old morphology, and therefore there is a reasonable expectation that the low-level controllers learned with the initial morphology will be useful in the new morphology. In contrast, a situation where our technique may not be expected to work as well would be one where a module is used in a drastically different context; for example, pretraining a module as a leg for a crawling robot and later using it as a finger in a hand, which may require a very different range of motions even if it reuses the same parts. That said, our task transfer experiments show that the low-level controllers learned through our technique can be useful when transferring to a different but similar task. Even when the task is sufficiently different as to require different local controllers, e.g. the modules used to walk over a flat terrain are finetuned when transferring to a terrain with steps, using the pretrained controllers as a starting point can reduce the training cost.\n\n\n**Under \"Questions\":** \n\n> Will this method generalize to multi-agent RL setting where the modular policy might be more complex than controlling 1-DoF motor?\n\nTo clarify, each local module does control subassemblies with more than one DoF -- for instance, the leg module of the centipede controls 5 joints. However, we agree that multi-agent RL would likely require more complex low-level controllers. At an abstract level, the problem of multi-agent RL is quite similar to what we address in our work -- the multi-agent policy can be decomposed into a higher-level controller that coordinates lower-level controllers corresponding to the individual agents. So while we have not experimented in this setting, we envision that our method could be applied to learn an appropriate division of labor between the boss that coordinates multiple agents and the more complex modules.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jqEAt7XWN2", "reviewer": "Reviewer_5WVH", "summary": "This paper introduces a new framework designed to create modular controllers allowing for quicker adaptation of control strategies when building new robots with similar methodology. The MeMo framework employs a novel modularity objective optimized alongside standard behavior cloning loss through noise injection. Experiment results in locomotion and grasping environments, ranging from simple to complex robot morphologies, demonstrate that MeMo improves training efficiency compared to multiple baselines. Additionally, MeMo’s modular approach enhances both structure and task transfer capabilities.", "questions": "No specific questions to add.", "limitations": "This work has provided reasonable discussion over the limitation and no potential negative social impact of the work needs to be considered.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The proposed method utilizes noise injection to build meaningful modular controllers that suit the application scenario as described: transfer the controller to a robot with similar but not identical morphology. The proposed method is easy to understand and should be easy to implement\n* This work provides reasonable grounding of the proposed method in Sec 3.1\n* The experiment shows the proposed method outperforms all baselines in locomotion and manipulation tasks as shown in Figure 6/7, in terms of better sample efficiency and better final performance in some environments. \n* This work provides a decent ablation study on the noise injection to validate the component's importance. including replacing the noise injection with different forms of regulation. \n* The paper is well-written and easy to follow, a reasonable amount of technical details are included in the method and supplementary materials.\n* The analysis of the learned module in section 5.4 and Figure 15 looks interesting.", "weaknesses": "* The tasks evaluated are relatively simple, only basic locomotion tasks and quite simple manipulation tasks are considered. \n* All tasks involve only a rigid body, which could be simulated much faster with GPU with the recent simulator, which should deliver a much shorter training time compared to the reported time in the appendix. It’s a bit questionable whether the proposed method is necessary if training a new policy from scratch takes a short period.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the review and are glad that they find our method easy to understand and empirical evaluation thorough. Below we address the reviewer’s concerns and questions.\n\n**Under \"Weaknesses\":** \n\n> The tasks evaluated are relatively simple, only basic locomotion tasks and quite simple manipulation tasks are considered.\n\nAlthough relatively simple, our tasks enable us to extensively evaluate the structure transfer capabilities of our framework, which is the primary focus of our work.  \n\n> All tasks involve only a rigid body, which could be simulated much faster with GPU with the recent simulator, which should deliver a much shorter training time compared to the reported time in the appendix. It’s a bit questionable whether the proposed method is necessary if training a new policy from scratch takes a short period.\n\nWhile we agree that the training time can be faster with recent simulators, the final reward of policies trained from scratch are below that of MeMo in our structure transfer experiments, an issue that cannot be fixed by faster simulation. These advances in simulation would also benefit MeMo, allowing the Boss controller to be retrained more quickly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k7vg2X9uoK", "reviewer": "Reviewer_tVBX", "summary": "The MeMo framework presented in this paper proposes an innovative approach for enhancing the transferability of control systems across robots with varied morphologies by utilizing pre-trained, modular controllers. This method facilitates rapid adaptation to new robot designs by leveraging previously trained modules, thereby significantly improving training efficiency over conventional methods like graph neural networks and Transformers.", "questions": "1. The comparative analysis raises some questions regarding the fairness of the evaluation. Specifically, were the control modules of the compared methods also retrained on your newly designed robots, or was it only the master controller that was retrained? If the latter is the case, this might lead to an unfair comparison since these pre-trained models have not been directly exposed to the newly designed robot configurations prior to testing.\n2. The transition from simulation to real-world applications is not addressed. The paper would benefit from a discussion on the expected challenges when implementing these modules in actual robotic systems, such as dealing with hardware inconsistencies and environmental variations. For example, how does the framework handle discrepancies in the physical properties between different robot assemblies, such as variations in leg mechanics?", "limitations": "N/A", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The presentation looks good. The results presented are intriguing and demonstrate potential.", "weaknesses": "N/A", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and are glad that they find our results intriguing. Below we address the reviewer’s questions.\n\n\n**Under \"Questions\":** \n> The comparative analysis raises some questions regarding the fairness of the evaluation. Specifically, were the control modules of the compared methods also retrained on your newly designed robots, or was it only the master controller that was retrained? If the latter is the case, this might lead to an unfair comparison since these pre-trained models have not been directly exposed to the newly designed robot configurations prior to testing.\n\nFor MetaMorph, we finetune the Transformer and the joint modules, as in the original work. For the NerveNet baselines, while parts of the network are fixed, prior work has found that such weight fixing actually improves transfer performance [1]. In the case of NerveNet-Conv, we experiment with both fixing the joint modules and finetuning the entire architecture and find that performance improves by fixing the joint modules. \n\n[1] Blake et al. “Snowflake: Scaling GNNs to High-Dimensional Continuous Control via Parameter Freezing”\n\n> The transition from simulation to real-world applications is not addressed. The paper would benefit from a discussion on the expected challenges when implementing these modules in actual robotic systems, such as dealing with hardware inconsistencies and environmental variations. For example, how does the framework handle discrepancies in the physical properties between different robot assemblies, such as variations in leg mechanics?\n\nWe agree that sim-to-real transfer is an important problem; however, the goal of learning controllers that are robust to variations outside of simulation is orthogonal to our problem of learning modules that generalize to different morphologies. We note that our baselines, NerveNet and MetaMorph, also perform their experiments only in simulation. In our revised paper, we will extend our discussion of sim-to-real transfer as an important line of future work. At a high level, we would expect our framework to face the same challenges in adapting to the real world as standard RL policies, including discrepancies in simulated physics vs real dynamics and, as the reviewer mentioned, hardware inconsistencies and environmental variations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5a4lYyj4hp", "reviewer": "Reviewer_bS7d", "summary": "This manuscript presents a hierarchical controller for reinforcement learning-based control. In particular, given a robot that can be decoupled into a high-level controller and a few low level (joint level) controllers, the two modules are learned jointly via a behavior cloning objective. The proposed method further makes the high-level controller robust to noise from the output of the low-level controllers by adding noise to their outputs. The model experiments on a few morphologies (6/12-leg centipede, 6/10-leg worm, etc) to demonstrate its effectiveness over vanilla RL and NerveNet.", "questions": "1. Can the expert policy on 6-leg centipede, and 6-leg worm be executed on 12-leg centipede and the 10-leg worm? The same question applies to the manipulation setting (4-finger to 5-finger claw).\n2. How effective is the method at correcting the dynamics of the system? I.e. after pre-training on the 6-leg centipede, if two or more of the pre-trained joints failed (i.e. uncontrollable), how fast can it adapt to the new setting?", "limitations": "Present in the manuscript.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Presents a hierarchical controller that enables sample efficient policy learning under morphological changes \n2. Prior methods require training on many morphologies, then generalize to an unseen morphology. The presented method is trained on one morphology and then transferred to a new morphology.\n3. Results suggest that the hierarchical formulation provides improved sample efficiency over baseline methods", "weaknesses": "1. Opposing strength 2, especially in simulation, many existing works randomize over many morphologies to enable efficient transfer between them or to an unseen morphology. It is unsure how this method can be trained on a dataset that contains more than one morphology. \n2. For the experiments, the reviewer is uncertain how significant is the morphology change in affecting the dynamics of the system.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the review. Below we address the reviewer’s concerns and questions.\n\n**Under \"Weaknesses\":** \n\n> Opposing strength 2, especially in simulation, many existing works randomize over many morphologies to enable efficient transfer between them or to an unseen morphology. It is unsure how this method can be trained on a dataset that contains more than one morphology.\n\nOur work is designed for the use case where the user does not have available to them a large number of morphologies. On the one hand, if you do have many morphologies, you can easily extend our approach to train the modules across the different morphologies and you may even get better modules; on the other hand, some of our baselines which are explicitly designed for such a use case may then outperform our approach.\n\n\n> For the experiments, the reviewer is uncertain how significant is the morphology change in affecting the dynamics of the system.\n\nOne piece of evidence to demonstrate the difficulty of the structure transfer task is in Figure 12 (Appendix A.12), where we test the zero-shot generalization of NerveNet-Conv by fixing all weights on the 6 to 12 leg centipede transfer. If the change in morphology did not significantly affect the dynamics, we would expect the pretrained model to still perform relatively well. However, our experiments show that without finetuning, the NerveNet-Conv performance is much worse than before, showing that the coordination required at transfer time is significantly different from pretraining.\n\n**Under \"Questions\":** \n> Can the expert policy on 6-leg centipede, and 6-leg worm be executed on 12-leg centipede and the 10-leg worm? The same question applies to the manipulation setting (4-finger to 5-finger claw).\n\nIn both locomotion and manipulation, our expert policy is a monolithic neural network, so due to the mismatch in the input and output space, it cannot be executed on our transfer morphologies. In addition, NerveNet [1] has MLP baselines in which they transfer weight matrices between hidden layers and they find that these baselines perform worse than their approach.\n\n[1] Wang et al. “NerveNet: Learning Structured Policy with Graph Neural Networks”\n\n\n> How effective is the method at correcting the dynamics of the system? I.e. after pre-training on the 6-leg centipede, if two or more of the pre-trained joints failed (i.e. uncontrollable), how fast can it adapt to the new setting?\n\nWe are not entirely sure how to interpret the question, but if the reviewer means that there is a possibility that due to the narrow interface between the boss and modules, the boss is unable to have fine-grained control over the modules, we find that empirically the boss is still able to learn new dynamics during transfer time. For example, in the 6 to 12 leg centipede transfer, some of the legs in the 12 leg centipede have more limited range of motion than in the 6 leg to avoid collisions. Our results also show that our approach outperforms the baseline where we train the same modular architecture from scratch on the transfer morphology, even though the baseline has more flexibility to learn the appropriate dynamics.\n\nAlternatively, if the reviewer is referring to a scenario where during transfer time, some joints may fail to perform as expected even when given the correct control signal, we believe our framework would be able to adapt in this case as well. Our framework learns modules that represent control signals on a low-dimensional manifold with respect to the boss signal. Even though the manifold would change as a result of the pretrained joints behaving differently at transfer time, our pretrained modules still enable us to navigate along a low-dimensional manifold to infer the optimal coordination, enabling the Boss to be quickly adapted at transfer time.\n\nIf neither response addressed the reviewer’s question, we are happy to provide more clarification.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper tackles an interesting and challenging problem. The reviewers appreciated the replies, additional details, and additional experiments. Overall the proposed method is an important first step, however there is also quite some way to go to make it more general (e.g. for more complex systems, multi-agent setting). On the flip-side focusing on simpler systems allows for an in-depth analysis.
After the discussion all reviewers are in favor of accepting this paper.
Decision: Accept (poster)