MetaMorph: Learning Universal Controllers with Transformers
Authors: Agrim Gupta
Linxi Fan
Surya Ganguli
Li Fei-Fei
Venue: ICLR 2022 Poster (2022)
Keywords: RL
Modular Robots
Transformers

Abstract: Multiple domains like vision, natural language, and audio are witnessing tremendous progress by leveraging Transformers for large scale pre-training followed by task specific fine tuning. In contrast, in robotics we primarily train a single robot for a single task. However, modular robot systems now allow for the flexible combination of general-purpose building blocks into task optimized morphologies. However, given the exponentially large number of possible robot morphologies, training a controller for each new design is impractical. In this work, we propose MetaMorph, a Transformer based approach to learn a universal controller over a modular robot design space. MetaMorph is based on the insight that robot morphology is just another modality on which we can condition the output of a Transformer. Through extensive experiments we demonstrate that large scale pre-training on a variety of robot morphologies results in policies with combinatorial generalization capabilities, including zero shot generalization to unseen robot morphologies. We further demonstrate that our pre-trained policy can be used for sample-efficient transfer to completely new robot morphologies and tasks.

Published as a conference paper at ICLR 2022
METAMORPH: LEARNING UNIVERSAL CONTROLLERS WITH TRANSFORMERS
Agrim Gupta', Linxi Fan Surya Ganguli' Li Fei-Fei!
Stannord University. SStanford Institute for Human-Centered Artificial Intelligence 3 NNVIDIA Corporation
(agrim, sganguli, feifeill (eiifeili@@ttanoord...oo edu, linxi finxif@nvidia.co
ABSTRACT
Multiple domains like vision, natural language, and audio witnessing tremen- dous progress by leveraging Transformers for large scale pre-training followed by task specific fine tuning. In contrast, in robotics we primarily train a single robot for a single task. However, modular robot systems now allow for the flexible com- bination of general-purpose building blocks into task optimized morphologies. However, given the exponentially large number of possible robot morphologies, training a controller for each new design impractical. In this work, we pro- pose MetaMorph, Transformer based approach to learn universal controller over a modular robot design space. MetaMorph is based on the insight that robot morphology is just another modality which we can condition the output of a Transformer. Through extensive experiments we demonstrate that large scale pre- training on variety of robot morphologies results in policies with combinato- rial generalization capabilities, including zero shot generalization to unseen robot morphologies. We further demonstrate that pre-trained policy be used for sample-efficient transfer to completely new robot morphologies and tasks.
INTRODUCTION
The field of embodied intelligence posits that intelligent behaviours can be rapidly learned by agents whose morphologies are well adapted to their environment (Brooks, 1991; Pfeifer & Scheier, 2001; Bongard, 2014; Gupta et al., 2021). Based on this insight, a robot designer is faced with a predica- ment: should the robot design be task specific or general? However, the sample inefficiency of tabula rasa deep reinforcement learning and the challenge of designing a single robot which can perform wide variety of tasks has led to the current dominant paradigm of 'one robot one task'. In stark contrast, domains like vision (Girshick et al., 2014; He et al., 2020) and language (Dai & Le, 2015; Radford et al., 2018), which are not plagued by the challenges of physical embodiment, have witnessed tremendous progress especially by leveraging large scale pre-training followed by trans- fer learning to many tasks through limited task-specific fine-tuning. Moreover, multiple domains are witnessing confluence, with domain specific architectures being replaced by Transformers (Vaswani al., 2017), a general-purpose purpose architecture with no domain-specifir inductive biases.
How can we bring to bear the advances in large-scale pre-training, transfer learning and general- purpose Transformer architectures, to the field of robotics? We believe that modular robot systems provide natural opportunity by affording the flexibility of combining a small set of general-purper purpose building blocks into a upttmopte morphology. Indeed, modularity at the level of hardware is motif which is extensively utilized by evolution in biological systems (Hartwell et al., 1999; Kashtan & Alon, 2005) and by humans many modern engineered systems. However, prior works (Wang al., 2018) on learning policies that can generalize
et al., 2018; Chen et al., 2018; Sanchez-Gonzaler across different robot morphologies have been limited to: (1) manually constructed variations of a single few base morphologies, i.e. little diversity in the kinematic structure; (2) low complexity of control (< 7 degrees of freedom); (3) using Graph Neural Networks (Scarselli et al., 2008) based on the assumption that kinematic structure of the robot is the correct inductive bias.
In this work, we take a step towards a more challenging setting (Fig. 1) of learning a universal controller for modular robot design space which has the following properties: (a) generalization

---

Published as a conference paper at ICLR 2022
Moduler Robol Design Space
Zero-Shot Generalization
Varialione
Dynamica
ommoture
maaleria
Jain Preiraning
Dat', new morpholeggee. modile parancters
Kinematics
MelaMorph
Varlations
Tasks
Learning universal controllers. Given modular robot design space, our goal is to learn a controller policy, which can generalize to unseen variations in dynamics, kinematics, new morphologies and tasks. Video available at this project page.
Figure 1:
to unseen variations in dynamics (e.g. joint damping, armature, module mass) and kinematics (e.g. degree of freedom, morphology. module shape parameters) and (b) sample-efficient transfer to new morphologies and tasks. We instantiate the exploration of this general setting in the UNIMAL design space introduced by Gupta et al. (2021). We choose the UNIMAL design space as it contains a challenging (15 20 DoFFs) distribution of robots that can learn locomotion and mobile manipulation in complex stochastic environments. Learning a single universal controller for huge variety of robot morphologies is difficult due to: (1) differences in action space, sensory input, morphology, dynamics, etc. (2) given modular design space, not all robots are equally adept at learning a task, e.g. some robots might inherently be less sample-efficient (Gupta al., 2021).
To this end, we propose MetaMorph, a method to learn a universal controller for a modular robot design space, MetaMorph is based on the insight that robot morphology is just another modal- ity which we can condition the output of Transformer. MetaMorph tackles the challenge of differences in embodiment by leveraging a Transformer based architecture which takes as input a sequence of tokens corresponding to the number of modules in the robot. Each input token is created by combining proprioceptive and morphology information at the level of constituent modules. The combination of proprioceptive and embodiment modalities and large scale joint pre-training leads to policies which exhibit zero-shot generalization to unseen variations in dynamics and kinemat- ies parameters and sample-efficient transfer new morphologies and tasks. Finally, tackle the differences in learning speeds of different robots, we propose dynamic replay busfer bulancing to dynamically balance the amount of experience collection for a robot based on its performance.
In sum, our key contributions are: (1) we introduce MetaMorph to learn a universal controller for modular design space consisting of robots with high control complexity for challenging 3D locomo- tion tasks in stochastic environments; (2) we showcase that our learned policy is able to zero-shot generalize to unseen variations in dynamics, kinematics, new morphologies and tasks, which par- ticularly useful in real-world settings where controllers need to be robust to hardware failures; (3) we analyze the learned attention mask and discover the emergence of motor synergies (Bernstein, 1966), which partially explains how MetaMorph is able to control a large number of robots.
RELATED WORK
Prior works on learning control policies which can generalize across robot morphologies have pri- marily focused parametric variations of single (Chen et al., 2018) or few (2 - 3) robot types (Wang et al., 2018 8; Sanchez-Gonzaled et al., 2018 8; Huang et al., 2020; Kurin et al., 2021). For gener- alizing across parametric variations of a single morphology, various approaches have been proposed like using a learned hardware embedding (Chen et al., 2018), meta-learning for policy adaptation (Al-Shedivat et al., 2017; Ghadirzadeh et al., 2021). kinematics randomization (Exarchos et al., 2020), and dynamics randomization (Peng et al., 2018). In case of multiple different morpholo- gies, one approach to tackle the challenge of differences in action and state spaces is to leverage Graph Neural Networks (Scarselli et al., 2008; Kipf & Welling, 2017; Battaglia et al., 201 8). Wang et al. (2018); Huang et al. (2020) use GNNs to learn joint controllers for planar agents (< 7 DoFs). Blake et al. (2021) propose freezing selected parts of networks to enable training GNNs for single morphology but with higher control complexity. The usage of GNNs is based on the assumption that the robot morphology is good inductive bias incorporate into neural controllers, which can be naturally modelled by GNNs. Recently, Kurin et al. (2021) also proposed using Transformers for training planar agents. Our work differs from Kurin et al. (2021) in the diversity and scale of
2

---

Published as a conference paper at ICLR 2022
training robots, complexity of the environments, conditioning the Transformer morphological information, and showcasing strong generalization to unseen morphologies and tasks (see B.1).
Another closely related line of work is the design of modular robot design spaces and developing al- gorithms for co-optimizing morphology and control (Sims, 1994) within a design space to find task- optimized combinations of controller and robot morphology. When the control complexity is low, evolutionary strategies have been successfully applied to find diverse morphologies in expressive soft robot design spaces (Cheney et al., 2014; 2018). In the case of rigid bodies, Ha (2019); Schaff' et al. (2019); Liao et al. (2019) have proposed using RL for finding optimal module parameters of fixed hand-designed morphology for rigid body robots. For more expressive design spaces, GNNs have been leveraged to share controller parameters (Wang et al., 2019) across generations or develop novel heuristic search methods for efficient exploration of the design space (Zhao et al., 2020). In contrast to task specific morphology optimization, III et al. (2021) propose evolving morphologies without any task or reward specification. Finally, for self reconfigurable modular robots (Fukuda & Nakagawa, 1988; Yim et al., 2007), modular control has been utilized in both real (Rubenstein et al., 2014; Mathews et al., 2017) and simulated (Pathak et al., 2019) systems.
LEARNING A UNIVERSAL CONTROLLER
We begin by reviewing the UNIMAL design space and formulating the problem of learning uni- versal controller for a modular robot design space as a multi-task reinforcement learning problem.
3.1 THE UNIMAL DESIGN SPACE
An agent morphology can be naturally represented kinematic tree, directed acyclic graph, corresponding to a hierarchy of articulated 3D rigid parts connected via motor actuated hinge joints. The graph G == (V, E) consists of vertices V = {f1,-... ........ Vn) corresponding to modules of the design space, and edges Cij E E corresponding to joints between i; and vy. Concretely, in the UNIMAL (Gupta al., 2021) design space, each node represents component which can be one of two types: (1) a sphere parameterized by radius and density to represent the head of the agent and form the root of the tree: (2) cylinders parameterized by length, radius, and density to represent the limbs of the robot. Two nodes of the graph can be connected via at most two motor motor-actuated hinge joints (i.c. G a multi multi-graph), parameterized by joint axis, joint limits and a motor gear ratio.
3.2 JOINT POLICY OPTIMIZATION
The problem of learning a universal controller for set of K robots drawn from a modular robot de- sign space a multi-task RL problem. Specifically, the control problem for each robot is an infinite- horizon discounted Markov decision process (MDP) represented by tuple (S, R, H, r), where represents the set of states, represents the set of available actions, T(8441 (8t, at) rep- resents the transition dynamics, R(s. a) is a reward function, H is the horizon and 7 is the discount factor. At each time step, the robot & receives an observation sf takes an action af, and is given a reward rf. A policy Te(aī1ss lsf) models the conditional distribution over action of E A given state sf E S. The goal is to find policy parameters which maximize the average expected return across all tasks: R R= R Ek En n²r/. We use Proximal Policy Optimization (PPO) (Schulman et al., 2017), a popular policy gradient (Williams, 1992) method for optimizing this objective.
METAMORPH
Progress in model-free reinforcement learning algorithms has made it possible to train locomotion policies for complex high-dimensional agents from scratch, albeit with tedious hyperparamter tun- ing. However, this approach is not suitable for modular design spaces containing exponentially many robot morphologies. Indeed, Gupta et al. (2021) estimates that the UNIMAL design space contains more than 10 robots. Hence, learning a separate policy for each robot is infeasible. However, the modular nature of the design space implies that while each robot morphology is unique, is still constructed from the same set of modules and potentially shares subgraphs of the kinematic tree with other morphologies. We describe how MetaMorph exploits this structure to meet the challenge of learning a universal controller for different morphologies.
3

---

Published as a conference paper at ICLR 2022
Leart Fositin Embeddings
MA
Por modile tokko
Linear Projection
Morphology
controll
Transformer
Diocodar
Exterocephive Ooe.
Transformer
Erccnder
Figure 2: MetaMorph. We first process arbitrary robot by creating ID sequenee of tokens corresponding to depth first traversal of its kinematic tree. We then linearly embed each token which consists of proprioceptive and morphological information, add learned position embeddings and encode the resulting sequence of vectors via a Transformer encoder. The output of the Transformer concatenated with linear embedding of exteroceptive observation before passing them through a decoder to output per joint controller outputs.
4.1 FUSING PROPRIOCEPTIVE STATES AND MORPHOLOGY REPRESENTATION
To learn policies that can generalize across morphologies, we must encode not only proprioceptive states essential for controlling a single robot, but also morphological information. From a multi-task RL perspective, this information can be viewed as a task identifier, where each task corresponds to a different robot morphology, all drawn from the same modular design space. Hence, instead of learning policy which is agnostic to the robot morphology, we need to learn policy conditioned on the robot morphology. Consequently, at each time step t (we drop the time subscript for brevity), the robot receives an observation s² (s sp. s which is composed of the morphology repre- sentation (st)), ). the proprioceptive states (s/),, and additional global sensory information (s See S A.1 for detailed description of each observation type.
4.2 MORPHOLOGY AWARE TRANSFORMER
The robot chooses its action via stochastic policy (a (s{) where 0 are the parameters of a pair of deep neural networks: policy network that produces action distribution (Fig. 2), and critic network that predicts discounted future returns. We use Transformers (Vaswani et al., 2017) to parametrize both policy and critic networks described in detail below.
Encode. We make a distinction between how we process local and global state information. Con- cretely, let k = (st s4) where sf (s s²)). Since the number of joints between two modules can vary, we zero pad sf, to ensure that input observation vectors are In order to provide an arbitrary robot morphology input to the Transformer, we first create 1D sequence of local observation vectors by traversing the kinematic tree in depth first order starting at the root node (torso in case of the UNIMAL design space). We then apply single layer MLP inde- pendently to st to create a D dimensional module embedding. We also add learnable 1D position embeddings to the module embeddings to automatically learn positional information:
same size, i.e. of E R.NM
[o(st; W We);... d<st W.) + W pos:
mo
where oi-) is the embedding function, W. E MDD are the embedding weights, and W pos NDD are the learned positional embeddings. Note that in practice we pad the input sequence of local observation vectors for efficient batch processing of multiple morphologies.
Process. From the module embeddings described above, we obtain the output feature vectors as:
m'a = MSA (LN(m=-1)) + m6-1,
= 1.
my MLP 2)) + m'r.
- 1
(3)
where MSA is multi-head attention (Vaswani et al., 2017) and LN is Layernorm (Lei Ba al., 2016).
Decode. We integrate the global state information s consisting of high-dimensional sensory input from camera or depth sensors. Naively concatenating 8² and sf in the encoder has two downsides: (1) it dilutes the importance of low-dimensional local sensory and morphological information; (2)
4

---

Published as a conference paper at ICLR 2022
Flal terrain
Varisble terrain
Obstadle
Figure 3: Environments. We evaluate our method on locomotion tasks: Flat terrain, Variable terrain: consists of three stochastically generated obstacles: hills, steps and rubble, and Obstacles: cuboid shaped obstacles of varying sizes.
it increases the number of Transformer parameters due to an increase in the dimensionality of the input embedding (D). Instead, we obtain the outputs policy network as follows:
g = (s Wg),
u(s*)) @(m,, g: Wa),
To(a"1s*) N .E), (4)
where y(-) is 2-layer MLP with parameters W df.) is embedding function with Wa as the emhedding weights. The action distribution modeled as a Gaussian distribution with state- dependent mean u(s{)) and fixed diagonal covariance matrix Σ. Similarly, for the critic network, we estimate value for the whole morphology by averaging the value per limb.
4.3 DYNAMIC REPLAY BUFFER BALANCING
Joint training diverse morphologies is challenging as different morphologies are adept at per- forming different tasks. Consequently, some morphologies might be inherently better suited for the pre-training task. Let us consider two robots: (A) Robot A locomotes in a falling forward manner, i... robot A is not passively stable; (B) Robot B is passively stable. Especially with early termina- tion, robot A will keep falling during the initial phases of training, which results in shorter episode lengths, whereas robot B will have longer episode lengths. Hence, more data will be collected for robot B in the earlier phases of training, and in turn will lead to robot B learning even faster, thereby resulting in 'rich gets richer' training dynamic. However, our goal to ensure that all morpholo- gies have similar level of performance at the end of training entire distribution morphologies.
want to generalize across the
To address this issue, we propose a simple dynamic replay buffer balancing scheme. On-policy algorithms like PPO (Schulman et al., 2017) proceed itcrations that alternate between experience collection and policy parameter update. Let & be any performance metric of choice, e.g. normalized reward, episode length, success ratio, etc. Let 7 be the training iteration number. At each iteration, sample the 1t robot with probability Pk. given by:
E E
P
Σ = O8 + (0)
(5)
where E [0, 1] is the discount factor and the exponent 3 determines the degree of dynamic pri- oritization, with our performance metric. We determine Pş by replacing &, with 1000 maximum episode length.
corresponding to the uniform case. In practice, we use episode length as
E, where the numerator is the
EXPERIMENTS
In this section, evaluate method MetaMorph in different environments, perform extensive ab- lation studies different design choices, test zero-shot generalization to variations in dynamics and kinematics parameters, and demonstrate sample efficient transfer to new morphologies and tasks. For qualitative results, please refer to the video on our project website
hhttps://mettmmrph--icll thub. io/site/
5

---

Published as a conference paper at ICLR 2022
Flat terrain
Variable terrain
Obstacles
2000
3000
1500
Reward
2000
1000
1000
1000
500
0
6
8
10
0
2
8
10
0
2
4
8
10
Iterations (x107)
MetaMorph
MetaMorph-NPP
MetaMorph-NM
GNN
MLP
Figure 4: Joint pre-training of modular robots. Mean reward progression of 100 robots from the UNIMAL design space averaged over runs in different environments for baselines and ablations described in $ 5.2. Shaded regions denote standard deviation. Across all 3 environments, Meta- Morph consistently outperforms GNN, and is able match the average reward achieved by per morphology MLP baseline, an approximate upper bound of performance.
5.1 EXPERIMENTAL SETUP
We create training set of 100 robots from the UNIMAL design space (Gupta al., 2021) (see $ A.2). We evaluate MetaMorph on three different environments (Fig. 3) using the MuJoCo sim- ulator (Todorov et al., 2012). all 3 environments the goal of the agent is maximize forward displacement over the course of an episode which lasts 1000 timesteps. The 3 environments are: (1) Flat terrain (FT); (2) Variable terrain (VT): VT an extremely challenging environment as during each episode new terrain is generated by randomly sampling sequence of terrain types and inter- leaving them with flat terrain. We consider types of terrains: hills, steps, and rubble; (3) Obstacles: cuboid shaped obstacles of varying sizes on flat terrain.
We use a dense morphology independent reward function all our tasks as not feasible to design reward function tailored to each morphology. In all tasks, reward function promotes forward movement using small joint torques (the latter obtained via a small energy usage penalty). In addition, described in $4.3, we use early termination across all environments when we detect fall (i.e. the torso height drops below 50% (FT, Obstacles) or 30% (VT) of its initial height).
5.2 BASELINES AND ABLATIONS
Baselines: We compare against the following baselines:
Variable terrain
terrain
Obstacles
(1) GNN: We modify the NerveNet model proposed by Wang et al. (2018) to learn control policies for complex 3D morphologies with variable number of joints. Specif- ically, we replace how the NerveNet model receives in- put and produces output by our encode and decode steps respectively ($4.2). In addition, the model receives the same observation as MetaMorph i.e. * = (st st st: and is trained with dynamic replay buffer sampling. Thus, the only difference is in the process step. This helps test if the domain-specific inductive bias of the robot kinematic tree in the GNN is necessary.
90
100
75
80
of
60
60
60
MetaMorph
w/o balancing
Figure 5: Importance of dynamic replay buffer balancing. We com- pare the percentage of robots with fi- nal performance greater than 75% of the ML.P baseline performance when trained jointly. Across all 3 environ- ments, on average for - 15% robots, MetaMorph w/o balancing is unable to leam a good control policy.
(2) MLP: We train all 100 robots separately with 2-layer MLP and report the average performance. This baseline serves approximate upper bound for our method.
We also do an ablation study of different de- sign choices involved in method. We refer our full method as MetaMorph and consider the following abla- tions: (1) MetaMorph-NPE: no learned position embeddings; (2) MetaMorph-NM: only provide
Ablations:

---

Published as a conference paper at ICLR 2022
Flat terrain
Reward
2000
Variable terrain
Reward
1000
Obstacles
Reward
1000
Armature
Density
Damping
Gear
Module param.
Joint angle
MetaMorph
MetaMorph-NPE
MetaMorph-NM
GNN
Figure 6: Zero-shot generalization. We create 400 new robots for each type of variation in dynam- ies parameters (armature, density, damping, gear) and kinematics parameters (module shape, joint angle). Bars indicate average zero-shot reward over 10 trials for 400 robots and error bars denote 95% bootstrapped confidence interval. Across all types of variations, and environments we find that MetaMorph considerably outperforms GNN, and MetaMorph-NM.
st = (s sy) as inputs, i.e. the model does not have access to information about the robot morphol- ogy.
Fig. shows learning curves across environments for training 100 morphologies. In all environ- ments MetaMorph can successfully match the average reward achieved via the per morphology MLP baseline on both FT, and obstacle environment. While MetaMorph performance in VT is slightly below the MLP baseline, note that it has not saturated and stopped training at 10 iterations across all three environments. Moreover, MetaMorph significantly more sample-efficient (5x) than training independent MLPs (5 10 iterations per robot). The GNN baseline saturates level 2 to 3 times below MetaMorph. In GNN based models, locality and neighborhood connectiv- ity explicitly baked into the model. Interestingly, just like ViT (Dosovitskiy al., 2021) sparingly utilizes the 2D neighborhood structure of images at the beginning of the model by cutting the image into patches, MetaMMoph uses the graph structure of the robot in the beginning by creating ID sequence corresponding to the kinematic tree by traversing the graph in depth first order. More- over, the position embeddings carry no information about the graph structure and are learned from scratch. We highlight that the learned position embeddings significantly improve the performance of MetaMorph, just as they do in Transformer based image classification. Finally, without access to the morphological information, MetaMorph-NM fails to learn policy that can control diverse robot morphologies. All of this substantiates our central claim that morphological state information necessary to learn successful control policies, although the kinematic graph need not be explicitly baked into neural architectures to learn policies capable of controlling diverse robot morphologies. Finally, test the importance of dynamic replay buffer balancing in Fig. 5, and find that balancing is necessary to learn a good control policy in 10 15% of robots across all 3 environments.
5.3 ZERO-SHOT GENERALIZATION
Our focus in this work is to learn policies that can generalize unseen robots drawn from a modular robot design space. In this section, demonstrate that MetaMorph shows favorable generalization properties across many different kinematic and dynamic variations.
Experimental Setup. For each of the 100 training robots, create dedicated test set test zero- shot transfer performance across two types of variations: dynamics (armature, density, damping, and motor gear) and kinematics (module shape parameters like radius, and length of cylindrical limbs, and joint angles). For each training robot, we randomly create different variants for each property, i.e. 400 robots with armature variations, and on. While creating variant, change the

---

Published as a conference paper at ICLR 2022
Flat terrain
Variable terrain
Obstacles
1500
3000
1500
Reward
1000
2000
1000
1000
500
500
Finetune Scratch
0
2
Iterations (x10²)
Figure 7: Fine tuning: New robot morphologies. Comparison of reward progression of 100 new robot morphologies averaged over 3 runs for pre-trained MetaMorph in the same environment vs from scratch. Shaded regions denotes standard deviation. Across all environments pre-training leads to strong shot performance and 2 3x more sample efficiency.
Obstacles (cylindors)
Escape
Obstacles Icylinders) 2k
2000
1500
Reword
500
Finetune Scratch
0-
(x10')
Figure Fine tuning: New robot morphologies and tasks. Left: Test environments. Right: Comparison of reward progression of 100 test robots averaged over 3 runs for pre-trained Meta- Morph (VT + Escape, Obstacles - Obstacles (cylinders)) vs from scratch. Shaded regions denotes standard deviation. Across all environments pre-training leads to strong zero-shot performance and 2 - 3x savings training iterations to achieve the same level average reward.
relevant property of all modules or joints. See Table for sampling ranges. We then compare zero-shot performance averaged over 10 trials.
Dynamics. First, consider generalization to different dynamics (Fig. 6). We find consistently that MetaMorph performs significantly better than MetaMorph-NM and GNN across all types of dynamic variations and all environments. In fact, the difference is nounced for harder tasks like VT and Obstacles. We note that this result is surprising as we do not do dynamies randomization during training, i.e., all robots in the training set have the same armature and damping parameters. Despite this, we see strong generalization performance.
Generalization:
pro-
Kinematies. Next, consider generalization different kinematics parameters (Fig. 6). This is significantly more challenging setting as the model has to generalize to unseen variations in module shape parameters and changes to joint angle ranges. In fact, changes to joint angles can significantly alter the range of possible motion and might necessitate different gait for successful locomotion. Consequently, we find that even though MetaMorph exhibits strong generalization performance compared to MetaMorph-NM and GNN in all 3 environments, there is indeed performance drop for the challenging setting of variations in joint angles. However, the zero-shot performance is encouraging and motivates our next set of experiments transfer learning.
Generalization:
5.4 SAMPLE EFFICIENT TRANSFER LEARNING
Experimental Setup. Here create 100 robots from the UNIMAL design space which were not part of the training set, and we demonstrate that MetaMorph shows favorable sample efficient transfer to unseen morphologies, and even unseen morphologies performing novel tasks.
Different morphologies. We first consider sample efficient learning of controllers for new mor- phologies the same task by taking a pre-trained MetaMorph model fine tuning it to learn to control morphologies the test set. In Fig. we compare the number of training iterations required to achieve particular performance level when fine tune MetaMorph
environment, and then
8

---

Published as a conference paper at ICLR 2022
1.75
MMM
1.5
1.25
300
350
400 Timesteps
450
500
300
400
Timesteps
0910
1.40
140
Figure 9: Emergent motor synergies. We plot the stable rank of the attention matrix (sr(Ar) ) for two agents performing locomotion on a flat terrain. sT(Ar) is small and oscillates between two values which correspond attention maps where groups of limbs are activated simultaneously (denoted by dark columns), characteristic signature of motor synergies.
vs training from scratch on the test set. Across all 3 environments we not only find strong zero-shot performance but also 2 times higher sample efficiency compared to training from scratch.
Different morphologies and novel tasks. Finally, we consider the most realistic and general setting mimicking the promise of modular robots, where we are faced with a novel task and want use new robot morphology which may be suited for this task. We consider the same set of 100 test robots on two new tasks (Fig. 8): (1) Escape: The agent starts at the center of bowl shaped terrain surrounded by small hills (bumps), and has to maximize the geodesic distance from the start location (escape the hilly region). (2) Obstacles (cylinders): cylinder shaped obstacles varying sizes (the size distribution is also different from the train task i.c. cuboid shapes). We transfer the learned policy from VT and Obstacles to Escape and Obstacles Cisiccs--ylideerss respectively. Again, we find that there is a strong zero-shot performance across all 3 environments and fine-tuning is 2 to times more sample efficient than training from scratch.
5.5 EMERGENT MOTOR SYNERGIES
We next search for potential explanation for how MetaMorph can coordinate the large number of DoFs (~ 16 100) across several agents. Our hypothesis is inspired by Bernstein (1966), which proposed the existence of muscle synergies as a neural strategy to simplify the control multiple DoFs by the central nervous system. A muscle synergy corresponds the constrained movements of sets of joints (or muscles) through co-activation by single neural signal. Such synergies obviate the need control all joints independently, by coupling sets of joints into adaptive functional groups
Although the definition of synergies (Bruton & O'Dwyer, 2018) vary in the literature, dimension- ality reduction is generally accepted signature of synergistic control (Todorov & Ghahramani, 2004; Todorov, 2004). Consequently, to test this hypothesis, in Fig. 9 we plot the stable rank of the attention matrix. For attention matrix A g for layer the stable rank is defined as: sr(A,) TA
A = Σ of where a; are the singular values of A, We note that sr(Ar) is small mmax"
and oscillates between values which correspond attention maps where groups of limbs activated simultaneously (denoted by dark columns), a characteristic signature of motor synergies. Hence, MetaMorph simplifies the control problem by learning to activate different motor synergies depending on both st and sp.
6 CONCLUSION
In this work, we explored how we can learn a universal controller for a large modular robot design space. To this end, we proposed MetaMorph, Transformer approach based on the insight that robot morphology is just another modality showcased that pre-training on a large collection of diverse morphologies leads to policies which can generalize unseen variations in kinematics, dynamics, our work serves as a step towards realizing the potential of large-scale pre-traee pre-training and fine-tuning in the field of robotics, paradigm that has seen tremendous success in vision and language.
which we can condition the output of Transformer. We
morphologies, and tasks. We hope that

---

Published as a conference paper at ICLR 2022
7 ACKNOWLEDGEMENT
We gratefully acknowledge the support by Department of Navy awards (N00014-16-1-21227, N00014-19-1-2477) issued by the Office of Naval Research.
REPRODUCIBILITY STATEMENT
We have released PyTorch (Paszke et al., 2019) implementation of MetaMorph GitHub (hotps: / //g.t thub. com/ commagrr// a 92 / metamorph). In addition, the repository also in- cludes all the robots and environments used for benchmarking.
REFERENCES
Maruan Al- Al-Shedivat, Trapit Bansal, Yuri Burda, Ilya Sutskever, Igor Mordatch, and Pieter Abbeel. Continuous adaptation via meta-learning in nonstationary and competitive environments, arXiv preprint arXiv: 011007000301111 2017.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint
2018.
Nikolai Bernstein. The co-ordination and regulation of movements. The co-ordination and regula- tion of movements, 1966.
Charlie Blake, Vitaly Kurin, Maximilian Igl, and Shimon Whiteson. Snowflake: Scaling gnns to high-dimensional continuous control via parameter freezing. arXiv preprint arXiv:2103.01009, 2021.
Josh Bongard. Why morphology matters. The horizons of evolutionary robotics, 6: 12-2152, 2014.
Rodney Brooks. New approaches to robotics. Science, 253(5025); 1991.
Michaela Bruton and Nicholas O'Dwyer. Synergies coordination: A comprehensive overview of neural, computational, and behavioral approaches. Journal of Neurophysiology, 120(6):2761- 2774, 2018
Tao Chen, Adithyavairavan Murali, and Abhinav Gupta. Hardware conditioned policies for multi- robot transfer learning. In NIPS, 2018.
Nick Cheney, Robert MacCurdy, Jeff Clune, and Hod Lipson. Unshackling evolution: Evolving soft robots with multiple materials and a powerful generative encoding. SIGEVOlution, 7(1): 11-23, August 2014. doi: 10. 01455//66133.2661777 2661735. 2661737.
Nick Cheney, Josh Bongard, Vytas SunSpiral, and Hod Lipson. Scalable co-optimization of mor- phology and control embodied machines. Journal of The Royal Society Interface, 15(143): 20170937, 2018.
Andrew M Dai and Quoc V Le. Semi-supervised sequence learning. Advances in neural information processing systems, 28:3079-3087, 2015.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Ioannis Exarchos, Yifeng Jiang. Wenhao Yu, and C Karen Liu. Policy transfer via kinematic domain randomization and adaptation. arXiv preprint arXiv:201 2020.
Toshio Fukuda and Seiya Nakagawa. Dynamically reconfigurable robotic system. In ICRA, pp- 1581-1586. IEEE, 1988.
10

---

Published as a conference paper at ICLR 2022
Ali Ghadirzadeh, Xi Chen, Petra Poklukar, Chelsea Finn, Mårten Björkman, and Danica Kragic. Bayesian meta-learning for few-shot policy adaptation across robotic platforms. arXiv preprint arXiv:2103.03697, 2021.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for ac- curate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 580-587, 2014.
Agrim Gupta, Silvio Savarese, Surya Ganguli, and Fei-Fei. Embodied intelligence via learning and evolution. Nature communications, 12(1))5721, 2021.
David Ha. Reinforcement learning for improving agent design. Artificial life, 25(4):352-365 2019.
Leland Hartwell, John J Hopfield, Stanislas Leibler, and Andrew W Murray. From molecular to modular cell biology. Nature, 402(6761):C47-C52, 1999.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pp. 9729-9738, 2020.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Thirty-Second AAAI Conference Artificial Intel- ligence, 2018.
Wenlong Huang, Igor Mordatch, and Deepak Pathak. One policy to control them all: Shared modular policies for agent- agnostic control. In ICML, pp. 4455 4464. PMLR, 2020.
Donald Joseph Hejna III, Pieter Abbeel, and Lerrel Pinto. Task-agnostic morphology evolution. In International Conference Learning Representations, 2021. URL net / forum? i CGQ6ENUMX6.
Nadav Kashtan and Uri Alon. Spontaneous evolution of modularity and network motifs. Proceed- ings of the National Academy of Sciences, 102(39)) 2005.
Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. ArXiv, hss/1609.02900) 2017.
Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik. Rma: Rapid motor adaptation for legged robots. arXiv preprint arXiv:2107:04034 2021.
Vitaly Kurin, Maximilian Igl. Tim Rocktäschel, Wendelin Boehmer, and Shimon Whiteson. My body is cage: the role of morphology in graph-based incompatible control. In ICLR, 2021.
Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47), 2020.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer Normalization. arXiv e-prints, art. arXiv:16077 nee.1.607006544 06450, July 2016.
T. Liao, G. Wang, B. Yang, R. Lee, K. Pister, S. Levine, and R. Calandra. Data-efficient learning of morphology and controller for microrobot. In 2019 International Conference on Robotics and Automation (ICRA), pp. 2488-2494 2019, doi: D0.110//CRA..219.88999
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei- Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín-Martin. What matters in learning from offline human demonstrations for robot manipulation. In 5th Anmual Conference Robot Learning, 2021. URL httttps://opereeviii wtwp://rttttteee////m
forum? dd-
Nithin Mathews, Anders Lyhne Christensen, Rehan O' O'Grady, Francesco Mondada, and Marco Dorigo. Mergeable nervous systems for robots. Nature communications, 8(1):1-7, 2017.

---

Published as a conference paper at ICLR 2022
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vito,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buce E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019.
Deepak Pathak, Christopher Lu, Trevor Darrell, Phillip Isola, and Alexci A Efros. Learning to control self-assembling morphologies: A study of generalization via modularity. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Buee E. Fox, and R. Gamett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Xue Bin Peng, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference robotics and automation (ICRA), pp. 3803-3810 IEEE, 2018.
Rolf Pfeifer and Christian Scheier. Understanding intelligence. MIT press, 2001.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training (2018), 2018.
Michael Rubenstein, Alejandro Comnejo, and Radhika Nagpal. Programmable self-assembly in thousand-robot swarm. Science, 345(6198):7957755 2014.
Alvaro Manchez-Gonzzalee Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin Riedmiller, Raia Hadsell, and Peter Battaglia. Graph networks as learnable physics engines for inference and control. In ICML, pp. 4479. PMLR, 2018.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE transactions on neural networks, 20(1):61-80, 2008.
Charles Schaff, David Yunis, Ayan Chakrabarti, and Matthew R Walter. Jointly learning construct and control agents using deep reinforcement learning. ICRA, pp. 9798-9805 IEEE, 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv e-prints, art. 1eXivv1707.063111 July 2017.
Karl Sims. Evolving 3d morphology and behavior by competition. Artificial life, 1(4):353-372, 1994.
Emanuel Todorov. Optimality principles in sensorimotor control. Nature neuroscience, 7(9):907~ 915, 2004.
Emanuel Todorov and Zoubin Ghahramani. Analysis of the synergies underlying complex hand manipulation. In The 26th Anmual International Conference of the IEEE Engineering in Medicine and Biology Society, volume 2, pp. 4637-4640. IEEE, 2004.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention all you need. NIPS, 2017.
Tingwu Wang, Renjie Liao, Jimmy Ba, and Sanja Fidler. Nervenet: Learning structured policy with graph neural networks. In ICLR, 2018
Tingwu Wang, Yuhao Zhou, Sanja Fidler, and Jimmy Ba. Neural graph evolution: Automatic robot design. In ICLR, 2019.
Ronald J Williams.
Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):22--256. 1992.
12

---

Published as a conference paper at ICLR 2022
Mark Yim, Wei-Min Shen, Behnam Salemi, Daniela Rus, Mark Moll, Hod Lipson, Eric Klavins, and Gregory S Chirikjian. Modular self selfffecofii robot systems [grand challenges of robotics]. IEEE Robotics & Automation Magazine, 14(1):43-52, 2007.
Allan Zhao, Jie Xu, Mina Konaković-Lukovic Josephine Hughes, Andrew Spielberg, Daniela Rus, and Wojciech Matusik. Robogrammar: graph grammar for terrain-optimized robot design. ACM Transactions on Graphics (TOG), 39(6):1-16. 2020.
13

---

Published as a conference paper at ICLR 2022
A IMPLEMENTATION DETAILS
In this section, we provide additional implementation details for our method.
A.1 INPUT OBSERVATIONS
In our setup, at each time step & (we drop the time subscript for brevity) the robot receives an observation s* = (sm sy. sg)) which is composed of the morphology representation (s the pro- prioceptive states (s*) and additional global sensory information (s*). We note that prior work (Kurin et al., 2021; Huang al., 2020) has defined morphology the connectivity of the limbs. However, connectivity is only one aspect of morphology, and in general a substantial amount of additional information may be required to adequately describe the morphology. Consider a modular robot composed of n E 1...Nr modules. For each module s consists of: (1) Module Parame- ters: Module shape parameters (e.g. radius, height), material information (e.g. density), and local geometric orientation and position of the child module with respect to the parent module. (2) Joint Parameters: This consists of information about joint type (e.g. hinge) and its properties (e.g. joint range and axis), and actuator type (e.g. motor) and its properties (e.g. gear). All this information can be found for example, in the Universal Robot Description Format (URDF) or in simulator specific kinematic trees (e.g, MuJoCo XML (Todorov et al., 2012)).
Similarly for each module s4. consists of the instantaneous state of the system: (1) Module Pro- prioception: 3D Cartesian position, 4D quaternion orientation, 3D linear velocity, and 3D angular velocity. (2) Joint Proprioception: Position and velocity in the generalized coordinates. Except the root node, each module will be connected to its parent module via set of joints. We adopt the con- vention of providing joint parameters and proprioception information to the child node. We note that the importance of proprioceptive observations has also been observed in learning good locomotion (Lee et al., 2020; Kumar et al., 2021) and manipulation policies (Mandlekar et al., 2021).
In general, additional global sensory information (s can be camera or depth sensor images. To save computation, we provide the information about the terrain as 2D heightmap sampled on a non- uniform grid. The grid is created by decreasing the sampling density as the distance from the root of the body increases. All heights are expressed relative to the height of the ground immediately under the root of the agent. The sampling points range from 1m behind the agent to 4m ahead of along the direction of motion, as well as 4m to the left and right.
A.2 ENVIRONMENTS
All our training and testing environments are implemented in MuloCo (Todorov et al., 2012). For a detailed description about the environments and reward functions please refer Gupta et al. (2021). Gupta et al. (2021) evolved UNIMALS in 3 different environments: flat terrain, variable terrain and manipulation in variable terrain. For each environment at the end of evolution, they had 100 task optimized robots. Out of these 300, we choose subset of 100 UNIMALS as our train set. We ensure that no robot has the exact same kinematic tree or kinematic parameters. Similarly, we created created the variants as
test set of 100 UNIMALS. Finally, for zero shot evaluation experiments described in $ 5.3. For creating kinematic variations for joint angles, we randomly selected a joint range for each joint from Table 2 which had alteast 50% overlap with the original joint angle range. This helped in preventing robot variants which could not be trained.
A.3 TRAINING HYPERPARAMETERA
We use Transformers (Vaswani et al., 2017) to parametrize both policy and critic networks. Global sensory information is encoded using 2-layer MLP with hidden dimensions [64, 64). We use Proximal Policy Optimization (Schulman et al., 2017) for joint training of agents in all environments. All hyperparameters for Transformer and PPO are listed Table 1. In addition, we use dynamic replay buffer sampling with = 0.1 and 3 1.0. For all our ablations and baselines, we use the same byperparameters. For MLP baseline we use a 2-layer MLP with hidden dimensions [64, 64] and for GNN baseline training of modular robots are shown in Algorithm 1. We ensure that all baselines and ablations have approximately the same number of parameters (~ 3.3 Million).
5-layer MLP with hidden dimension 512. The details of the joint
14

---

Published as a conference paper at ICLR 2022
A.4 EVALUATION METHODOLOGY
In general, the performance of RL algorithms is known to be strongly dependent on the choice of seed for random number generators (Henderson al., 2018). Hence, ablation experiments for 3 random seeds. We found that the training curves were robust to the choice of seeds as indicated by small standard deviation (Fig. 4, 10). Consequently, we evaluate the zero shot performance using pre-trained model corresponding to single seed (Fig. 6). For our transfer learning experiments (Fig. 7, 8), we fine tune the model with the random seed corresponding to the one used for pre-training (for all 3 seeds).
all our baseline and
B ADDITIONAL EXPERIMENTS
B. BASELINES AND ABLATIONS
Baselines: We compare against the following baselines:
(1) MetaMorph-AO: Direct comparison to Amorpheus (Kurin et al., 2021) is not feasible due to the following reasons: (1) does not work with 3D morphologies with variable number of joints between limbs; (2) does not incorporate exteroceptive observations; (3) Amorpheus ensures balanced col- lection of experience by sequentially collecting data on each morphology and maintaining separate replay buffer per morphology. Further, updates to policy parameters This sequential nature of the algorithm is not amenable to scaling, and would require ~ 30 GPU days to train for 100 million iterations on Nvidia RTX 2080, while our method only needs 1.5 GPU days (~ 20x training speedup).
also performed sequentially.
Hence, compare with MetaMorph-AO (Amorpheus Observation) the closest variant to Amor- pheus, where we provide the same input observations as described Kurin et al. (2021). Specifi- cally, we provide the following input observations: one hot encoding of unique limb ID,, 3D Carte- sian position, 4D quaternion orientation, 3D linear velocity, and 3D) angular velocity, position in generalized coordinates, and normalized joint angle ranges. Note that although both MetaMorph and Amorpheus don't I explicitly incorporate the graph structure as input to the policy, our input ob- servations include additional morphology information (see $ B. In contrast, except for joint angle ranges, Amorpheus does not incorporate morphology information.
(2) Multi-Taan Task MLP: We train 6- layer MI.P with hidden dimension of 512. The MI. receives the same observations as MetaMorph, and has the same number of parameters.
Flat Terrain
GNN
3000
MT-MLP
Reward
MetaMorph
2000
MetaMorph-AO
Ablations: We perform additional
MetaMorph-NPe
ablation studies to understand the im- portance of learnt position embed- dings and morphology information in input observation. We refer our full method as MetaMorph and con- sider the following additional ab- lations: (1) MetaMorph-NMT (No Morphology information + Task en- coding): we only provide s (s), st) as inputs, i.e. the model does not have access to information about the robot morphology. In addition, we provide a context token with binary encoding to iden- tify the morphology; (2) MetaMorph-HPE (Hand-designed Position Encoding): we provide s (sf ck p: st)) as inputs, with s containing a per limb unique ID.
1000
MetaMorph-HPS
MetaMorph-NM
0
MetaMorph-NMT
0
2
4
6
8
10
Iterations (x10?)
Figure 10: Baselines and ablations. Mean reward progres- sion of 100 robots from the UNIMAL design space aver- aged over 3 runs in flat terrain for baselines and ablations described in $ 5.2 and B. Shaded regions denote stan- dard deviation.
Fig. 10 shows the learning curves for joint training in the flat terrain environment for 100 training morphologies. Multi-task MLP baseline struggles to learn good policy, which we attribute to the difficulty of the task due to the rich diversity of morphologies. We next investigate the role of learn position embeddings and morphological information. Meta.Mrrp- performs slightly better than MetaMorph-NPE, indicating that adding unique limb ids improves performance. However, there is
15

---

Published as a conference paper at ICLR 2022
2
0.8
0.6
0.4
11 9 8
0.2
2 4 7 1011
1011
1011
Figure II: Position embeddings. Cosine similarity position embeddings of MetaMorph for different seeds in flat terrain environment.
still a large gap between MetaMorph and MetaMorph-HPE, suggesting that learnt position embed- dings capture additional information. Although MetaMorph-AO performs better than MetaMorph- NM due to the addition of unique limb ids and joint angle ranges, it is still significantly worse than MetaMorph as does not incorporate morphological information. Finally, MetaMorph-NMT also performs poorly due to the lack of morphological information.
B.2 POSITION EMBEDDING
To better understand the information captured by position embeddings we visualize their cosine similarity. We found that across all seeds and in all environments (we only show flat terrain for brevity) diagonal pattern emerged (Fig 11). Although, this might suggest that learnt position embeddings are capturing unique position identifier, we found that manually specifying a unique limb identifier to perform much worse (see MctaMorph-HIE in Fig. 10). Thus, indicating that these embeddings are capturing additional useful information. We believe that further investigation needed to better understand the information captured by position embeddings and defer it to future work.
B.3 KINEMATIC TREE TRAVERSAL ORDER
We first process arbitrary robot by creating 1D sequence of tokens corresponding the depth first traversal of its kinematic tree. We note that the DES ordering is not unique, and for nodes with similar depth, adopted the convention of visiting the nodes which come first in the MuJoCo XML. This convention also adopted by MuJoCo when parsing the XML. We found that zero-shot transfer of the learnt policy was robust to an opposite ordering of nodes and the performance dropped by 75%. However, we found that the policy could be made robust to variations in the DFS ordering by training the model with a simple data augmentation strategy of randomizing the visitation order of nodes at the same depth.
C LIMITATIONS AND FUTURE WORK
In this work, we explored how we can learn a universal controller for a large modular robot design space. We make key assumption that we already had access large collection of robots which were optimized for the task of locomotion. Hence, currently we require two stage pipeline where we first create robots and then pre-train them jointly. An important direction of future work would be to combine these two phases in efficient manner. Moreover, although found that MetaMorph has strong zero generalization to unseen variations in kinematics and dynamics, there is indeed performance drop zero-shot generalization to morphologies. Hence, important avenue of exploration creating algorithms for sample efficient transfer learning. Finally, our current suite of tasks focuses primarily on locomotion skills. An important line of future work will involve designing general purpose controllers which could perform multiple skills.
16

---

Published as a conference paper at ICLR 2022
Algorithm MetaMorph: Joint Training Modular Robots
1: Input:
To: policy function
Vg: value function
R: replay buffer
K: training pool of robots
Ps: probability of sampling robot k
2: Initialize: Learnable parameters for æe. for Vg, uniform distribution
3: for i=1,2...NNee do
4: # Collect robot experience in parallel
5: for jol.,,_a... do
6:
Sample robot E K, according to Equation 5
7:
# Collect episode of experience
8:
=
{Se: 0t, 8t+1, (++1) ~ m
9:
# Add data the buffer
10:
R + R RU
end for
12:
# Update policy and value functions
for jor.j.......NN do
14:
Sample minibatch of data from R me < PPOUpdate(TT,, r)
16:
Ve < PPOUUpdat(V2..77
17:
end for
18:
# Sample all robots uniformly random initially
19: if >= Nwarmup then
20: P samplinggrrobUpdate((R)) according Equation 5
21: end if
22: end for
17

---

Published as a conference paper at ICLR 2022
Hyperparameter
Value
Discount yy
.99
GAE parameter
0.95
PPO clipping parameter
0.2
Policy epochs
Batch size
5120
Entropy coefficient
0.01
Reward normalization
Yes
Reward clipping
10. 10]
Observation normalization
Yes
Observation clipping
| 10. 10
PPO
Timesteps per rollout # Workers
2560
16
# Environments
32
Total timesteps
10
Optimizer
Adam
Initial learning rate
0.0003
Learning rate schedule
Linear warmup and cosine decay
Warmup Iterations
Gradient clipping (12 norm) Clipped value function
0.5
Yes
Value loss coefficient
0.5
Number of layers
Number of attention heads
Embedding dimension Feedforward dimension
128
Transformer
1024
Non linearity function Dropout
ReLU
0.1
Table Training hyperparameters.
Kinematics
Variation type
Value
Limb radius
[0.03, 0.05
Limb height
[0.15, 0.45]
[ 30, 0), (0, (-30033))
45, 45), (-45,0), 40),
Joint angles
-00 0), (0, 60), (-60, 60)
(-90, 0), (0, 90),
Dynamics
Armature
0.
Density Damping Gear
[0.8, 1.2]x limb density [0.01, 5.0)
[0.8. 1.2]x motor gear Table 2: Dynamics and kinematics variation parameters.
18


Reviews:
[{"review_id": "uywG2bltvLf", "reviewer": "Reviewer_XpGx", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I think this paper is very interesting. In my opinion, its main strengths are:\n1. The design changes to the standard transformer architecture are sound and interesting for the problem setup. The main idea, i.e. conditioning the architecture to the robot morphology, is sound and interesting. Also, the dynamic replay buffer (basically storing samples in function of performance) is a simple but very nice idea. None of those are really breakthroughs but they are sound technical contributions that could influence future robotics research.\n2. The experimental results are solid (even though some things could be improved, more on this later).\n3. Writing is good and easy to follow, with figures helping to understand the main concepts.\n\nThe following is a set of weaknesses that I think could improve the contribution. None are major weaknesses or flaws, but just recommendations to improve the current submission:\n\n1. The motivation to study the problem of a universal controller for robot morphologies could be improved. It looks like the main motivation is to \"bring the advantages of pre-training into robotics\", which sounds more like a methodology than a research problem. I think a more solid connection to real-world problems could be the situation when some parts of the robot fail (a propeller for a drone, a joint for a manipulator, etc.), and the control policy should be robust to these changes. The robustness to morphologies in the same training distribution could be very valuable there I think.\n2. Several prior works on legged locomotion ( like R1 or R2) condition the control policy on some environment and robot characteristics. This is in a way similar to the proposed approach of conditioning the policy on morphology, so I think it deserves mention.\n3. I did not fully get Eq. 5, which explains the dynamic replay buffering. Assuming $\\beta>0$ (which seems okay according to the text),  the longer the episode length, the higher the probability of being sampled. This is in contrast with the motivation of dynamic replay buffering balancing, as explained in the example in section 4.3, where one wants to sample robots with short episode length (thus difficult to train).\n4. I find it interesting that in the presence of obstacles, the meta-policy works much better than the per-robot MLP. Why is this the case? Why don't we observe the same in the flat and variable terrain?\n5. As baseline, it would be nice to have a naive approach that substitutes the transformer architecture with a simple MLP. This MLP would have the same observations of the transformer and the same number of parameters to make the comparison as fair as possible. This naive baseline could give more intuitions on the advantages of the transformers for the task.\n6. In Section 5.5, it would be nice to analysize a specific robot morphology and try to decode the resulting motor synergies. For example, would be nice to answer the question: why is limb N highly correlated to all other parts of the robot?\n7. Finally, it would be nice to have a section discussion limitations of the approach. The biggest now seems that the generalization to robot out of distribution (in particular with different kinematics) is limited, and 2x10^7 iterations or more might be required to fine-tune on a new robot (which is not really few-shot). Overall, it would be nice to be open about the limitations to facilitate future work.\n \n[R1] RMA: Rapid Motor Adaptation for Legged Robots, Kumar et al.\n\n[R2] Learning Quadrupedal Locomotion over Challenging Terrain, Lee et al.", "summary_of_the_paper": "This paper proposes a Transformer-based universal policy to control modular robots. Being the space of modular robots combinatorial, it is (almost) impossible to train specific policy for each sample of the \"robot distribution\". Therefore, the paper proposes a transformer-based architecture design with explicit information about the robot morphology that, trained on a large number of possible morphologies. This universal controller generalizes zero-shot to unseen robot designs belonging to the training distribution and can be efficiently fine-tuned on new robots and tasks. The approach is thoroughly evaluated on different tasks.", "summary_of_the_review": "The paper tackles an interesting problem in a sound way introducing some key technical contributions. Addressing some weaknesses in the presentation and in the experimental part, as well as being more open about limitations, could improve the quality of the submission.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "omjF3OA4wTe", "reviewer": "Reviewer_m7dm", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nOverall the paper is well written. It proposes an interesting way to use Transformers for learning a universal controller for different robot morphologies. The idea of a dynamic replay buffer is simple but exhibits improvement in learned behaviors. The results are thorough and cover most aspects of their approach and exhibit improved performance over prior methods. The ablation studies are also well-suited to describe the design choices. \n\nWeakness:\nThe contributions on algorithmic and network designs are marginal. Their approach is a direct application of Transformers and Reinforcement Learning approaches. \n\nClarification: The description of section 4.3 can be improved, it is slightly confusing to follow. It seems Eq 5 will give a higher probability to robots with longer episode lengths which is in contrast to their description. Please elaborate or clarify if I missed something.  \n", "summary_of_the_paper": "This paper presents MetaMorph, a Transformer based universal controller to learn behaviors across different robot morphologies. Their Transformer module takes a sequence of tokens as input, corresponding to the number of modules in the robot. Each input token comprises the proprioceptive and morphology information for each constituent robot’s module. MetaMorph is trained using a standard model-free Proximal Policy Optimization (PPO) method. However, to ensure all robots get to explore given environments and gather a similar amount of experience, the paper introduces a dynamic replay buffer which prioritizes robots’ selection for experience collection based on their previous iteration's episode lengths. The results are compared against MLP based experts trained for all individual robots, Graph Neural Network-based universal controller, and ablated modules of MetaMorph to highlight design choices. Moreover, results also demonstrate the generalization capacity of MetaMorph to different robots through zero-shot transfer and fine-tuning. \n", "summary_of_the_review": "It proposes an interesting way to use Transformers for learning a universal controller for different robot morphologies and presents an extensive set of experiments. However, the overall contributions to algorithms and network designs are limited.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UzG5WRXD1Es", "reviewer": "Reviewer_PnzQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is beautiful work! The approach is well-motivated, and the experiments are thorough, covering a wide diversity of robots and environments, and demonstrating remarkable generalization. On the whole, the paper is exceptionally well-written and informative.\n\nThe main weakness I see is a failure to adequately distinguish the proposed approach (MM) from the very similar AMORPHEUS approach (AM) by Kurin et al., 2021. The current work claims that MM conditions the transformer on morphological information, while AM does not. However, it seems to me that MM and AM condition the transformer on essentially the same information. The confusion seems to arise because the two papers use the term *morphology* in two separate ways. This requires careful explanation.\n\nEach robot consists of one body and some number of limbs branching from it. (This paper calls the body and limbs *modules*, but I’ll call them all limbs for ease of visualization.) The information about the robot’s limbs can be divided into 3 categories:  limb-to-limb connectivity, static limb state (like limb type and its joint’s range of motion), and dynamic limb state (like instantaneous angular position and velocity). For processing by the network, standard practice is to combine the state of each joint (both static and dynamic) with the state of the limb that it swings.\n\nAM (Kurin et al., 2021) is a modification of SMP (Huang et al., 2020). In all three architectures (MM, AM & SMP), observations include both static and dynamic state information of joints and limbs, but not explicit limb-to-limb connectivity information. SMP uses limb connectivity to define the computational graph over which messages are passed, from limb-to-limb. By contrast, MM and AM use a transformer encoder which passes messages between every pair of limbs, whether directly connected to each other in the robot or not. Kurin et al., 2021, demonstrate that AM’s all-to-all message passing performs better than SMP’s restricted message passing.\n\nNow to the terminology. Kurin et al., 2021, uses the term *morphology* to refer to the *connectivity* of the limbs. But *this* paper uses the term *morphology* to refer to the *static state* of limbs (or their joints). This seems to explain why the paper states that MM differs from AM in “conditioning the Transformer on morphological information”, as if AM didn’t. This is incorrect, because both AM and MM condition the transformer on both static and dynamic state, and neither one conditions the transformer on connectivity information. If AM leaves out some important piece of static information, or if there is some other essential difference between AM and MM, the paper in its current form does not make this difference clear.\n\nI agree that the current work differs from that of Kurin et al., 2021, “in the diversity and scale of training robots, complexity of the environments, … and showcasing strong generalization to unseen morphologies and tasks”. But I don’t see an important architectural difference between the models. (As the paper states, “the position embeddings carry no information about the graph structure”, so limb-to-limb connectivity is not carried by MM’s position embeddings.) AM is clearly the most meaningful baseline model to compare MM against, but this was not done, despite the availability of the AM code.\n\nThe experimental results show that MM’s position embeddings boost performance, but I find no explanation for why they do. Could it just be that they provide unique IDs to help the transformer differentiate the modules from each other, similar to a scheme employed by SMP?\n", "summary_of_the_paper": "The paper presents MetaMorph (MM), a transformer-based architecture for generalization over robot morphologies. Extensive experimentation demonstrates that policies learned by MM transfer well to new robot morphologies.", "summary_of_the_review": "The paper presents compelling experimental results, but does not adequately differentiate MetaMorph from AMORPHEUS by Kurin et al., 2021, and does not include AMORPHEUS as a baseline in the experiments.\n\n--- POST-DISCUSSION UPDATE ---\n\nThe authors have clarified the distinction between their work and that of Kurin et al., 2021, and have performed new experiments comparing MetaMorph to an ablated version (MetaMorph-AO) that approximates the more restricted morphological observations used by Amorpheus. \n\nAlthough MetaMorph and Amorpheus use differing sets of input features, they are still fundamentally the same architecture. (The \"morphology aware transformer\" used by MetaMorph is just a standard transformer encoder, like that of Amorpheus.) So in my view, there is little novelty in the Amorpheus architecture, and the work's value lies in the scope of its experiments. I have raised my evaluation accordingly.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PCzMpuQO7h", "reviewer": "Reviewer_Hxo2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Pros\n\n- Strong empirical results for multitask training/zero-shot generalisation and transfer.\n- Dynamic replay balancing is an idea that might be useful for multitask RL outside this particular setting.\n- The paper is mostly clearly written and has nice graphics.\n- Well-written related work section.\n\n### Cons\n\n- I find it confusing that the method name has 'meta' in it?\n- Lack of clear comparison with Amorpheus (Kurin et al, ICLR 2021) that also uses transformers for incompatible multitask control.\n- Dynamic replay balancing will only work if the environments are somewhat similar given the used metric (e.g. similar number of episode steps).\n- Section 5.5 is confusingly written. The relationship to muscle synergies sounds cool and sells well, but I don't think it makes a lot of sense to me in this setting or it requires more explanation.\n- I don't think that MetaMorph-NM is a fair ablation that tells how morphological information is useful to training. A fairer ablation would be providing a one-hot encoding of a task instead of some meaningful morphological information. Can you run this ablation during the rebuttal period?\n- Some of the very important details are omitted, i.e. what does $s^k_m$ contain? How do you learn positional embeddings? See more in Questions/Comments section.\n\n### Questions/Comments\n\n- Does NerveNet benefit from balancing? When you compare with GNN in Figure 4, does it include the balancing?\n- Amorpheus (Kurin et al, ICLR 2021) shows that zero-shot generalisation exhibits quite a large variance across the seeds, however, your approach does not suffer from this issue, why do you think this is the case?\n- When you study zero-shot/transfer behaviour, what is the experimental protocol? You run multiple seeds → select a model → run zero-shot evaluation/finetune for transfer. How do you select a model? Do you select a single model or one per each of the original training runs?\n- You don't compare to Amorpheus, why? What ablation in Figure 4 is the closest to Amorpheus?\n- W_{pos} should take a k-th row for the dimensions to make sense, not adding the full matrix.\n- How do you obtain a positional embedding matrix for your zero-shot transfer experiments?\n- DFS order depends on the internal representation of your tree, i.e. A→B←C might go to A or C after B depending on the order you store them in. What happens to your method if you do zero-shot generalisation transfer on the mirrored agent (so that DFS returns a different ordering of the same nodes)? Can you provide the results of this experiment?\n- How do you learn the positional encodings? How do they look like if you plot them as a matrix?\n- What does the morphological information contain, i.e. what is $s_m^k$?\n- typo on page 2 'planer' → 'planar'\n- Are Figure 9 results consistent across different seeds?\n- The related work section is missing a recent paper: \"*Snowflake: Scaling GNNs to high-dimensional continuous control via parameter freezing\", Blake et al, NeurIPS 2021*", "summary_of_the_paper": "The paper uses transformers for incompatible multitask continuous control and shows impressive empirical results on a wide range of tasks in multitask training, zero-shot generalisation and transfer. The authors propose morphology-based conditioning and dynamic replay buffer balancing to increase the performance.", "summary_of_the_review": "This is a good paper with impressive empirical results. While transformers have been applied for incompatible multitask continuous control before, none of the works scaled the setting that much. Dynamic replay balancing proposed in the paper might be useful in multitask RL in general. I would give this paper a 7, but the system does not allow it. I am putting 6 for now and am willing to increase the score if the authors address my questions/comments during the discussion period.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JUv5J-te6B", "reviewer": "Reviewer_rhAB", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- The problem formulation (especially 4.2 Morphology Aware Transformer) is novel and quite persuasive.\n- The proposed claim is appropriate in that robot morphology can be understood as a modality on which we can condition the output of a Transformer.\n- Implementation details are shared so as easy to reproduce.\n\nWeaknesses\n- In the 5 Experiments, the model sizes of the other design choices for comparison are smaller than the proposed model, which may lead to unfair comparison.", "summary_of_the_paper": "In robotics, researchers primarily train a single robot for a single task, while modular robot systems suggest the flexible combination of general-purpose building blocks into task optimized morphologies. The authors proposed a Transformer approach to learn a universal controller over a modular design space leveraging the “large-scale pre-training and fine-tuning” scheme.", "summary_of_the_review": "I generally agree with the argument of the paper. As the authors mentioned in the Abstract, Deep learning research fields like vision, natural language, and audio actively leverage Transformers for large-scale pre-training followed by task-specific fine-tuning. The idea makes sense that it deals with robot morphology as a modality on which we can condition the output of a Transformer in the modular robot design setting.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper aims to improve generalization and sample efficiency in robotic control. In this regard authors note that modular robot systems, which provide building blocks for a task-specific morphology, can be considered just another domain where transformers can be used. The authors propose to learn a universal controller over this modular design space and leverage successful “large-scale pre-training and fine-tuning” scheme for transformer. We thank the reviewers and authors for engaging in an active discussion. Reviewers found that methodological novelty is limited ("The contributions on algorithmic and network designs are marginal. Their approach is a direct application of Transformers and Reinforcement Learning approaches.") and there are very similar approaches in the literature (e.g., AMORPHEUS approach (AM) by Kurin et al., 2021.) Despite these shortcomings, overall the reviewers found the experimental results to be strong and analysis to be thorough, which will be valuable to the community.
Decision: Accept (Poster)