Transform2Act: Learning a Transform-and-Control Policy for Efficient Agent Design
Authors: Ye Yuan
Yuda Song
Zhengyi Luo
Wen Sun
Kris M. Kitani
Venue: ICLR 2022 Oral (2022)
Keywords: Agent Design
Morphology Optimization
Reinforcement Learning

Abstract: An agent's functionality is largely determined by its design, i.e., skeletal structure and joint attributes (e.g., length, size, strength). However, finding the optimal agent design for a given function is extremely challenging since the problem is inherently combinatorial and the design space is prohibitively large. Additionally, it can be costly to evaluate each candidate design which requires solving for its optimal controller. To tackle these problems, our key idea is to incorporate the design procedure of an agent into its decision-making process. Specifically, we learn a conditional policy that, in an episode, first applies a sequence of transform actions to modify an agent's skeletal structure and joint attributes, and then applies control actions under the new design. To handle a variable number of joints across designs, we use a graph-based policy where each graph node represents a joint and uses message passing with its neighbors to output joint-specific actions. Using policy gradient methods, our approach enables joint optimization of agent design and control as well as experience sharing across different designs, which improves sample efficiency substantially.  Experiments show that our approach, Transform2Act, outperforms prior methods significantly in terms of convergence speed and final performance. Notably, Transform2Act can automatically discover plausible designs similar to giraffes, squids, and spiders. Code and videos are available at https://sites.google.com/view/transform2act.

Published as a conference paper at ICLR 2022
TRANSFORM2ACT. LEARNING A TRANSFORM-AND- CONTROL POLICY FOR EFFICIENT AGENT DESIGN
Ye Yuan', Yuda Song', Zhengyi Luo', Wen Sun', Kris M. Kitani! Caamegic Mellon University, Cornell University {yyuan2, yudas, zluo2, kkitani ( (cs. cmu. edu, ws4550 c0030ecooeel....
Transform
Control
Transform2Act learns transform and-aanddrree policy that first applies transform actions to design an agent and then controls the designed agent to interact with the environment. The giraffe- like agent obtained by Transform2Act extremely fast and remain stable (see video).
Figure 1:
ABSTRACT
An agent's functionalityy is largely determined by its design, i.e., skeletal struc- ture and joint attributes (e.g., length, size, strength). However, finding the optimal agent design for a given function is extremely challenging since the problem is inherently combinatorial and the design space is prohibitively large. Additionally, it can be costly to evaluate each candidate design which requires solving for its optimal controller. To tackle these problems, our key idea is to incorporate the design procedure of an agent into its decision-making process. Specifically, we learn conditional policy that, in episode, first applies sequence of transform actions to modify an agent's 's skeletal structure and joint attributes, and then ap- plies control actions under the new design. To handle variable number of joints across designs, we use a graph-based policy where each graph node represents a joint and uses message passing with its neighbors to output joint-speeific actions. Using policy gradient methods, our approach enables joint optimization of agent design and control as well as experience sharing across different designs, which improves sample efficiency substantially. Experiments show that our approach, Transform2Acc, outperforms prior methods significantly in terms of convergence speed and final performance. Notably, Transform2Act can automatically discover plausible designs similar to giraffes, squids, and spiders. Code and videos are available https: //tttsstssss///////.////ee/ssses google com/view/ http//rrnsffooooo form2act..
INTRODUCTION
Automatic and efficient design of robotic agents in simulation holds great promise in complementing and guiding the traditional physical robot design process (Ha et al., 2017; 2018), which can labo- rious and cine-consuming. In this paper, consider setting where optimize both the skeletal structure and joint attributes (e.g., bone length, size, and motor strength) of an agent to maximize its performance the design space, i.e., all possible skeletal structures and their joint attributes. is prohibitively vast and combinatorial, which also makes applying gradient-based continuous optimization methods dif- ficult. Second, the problem is inherently bi-level: (1) we need to search an immensely large design space and (2) the evaluation of each candidate design entails solving a computationally expensive inner optimization to find its optimal controller. Prior work typically uses evolutionary search (ES) algorithms for combinatorial design optimization (Sims, 1994; Wang et al., 2019). During each it- eration, ES-based methods maintain large population of agents with various designs where each agent learns to perform the task independently. When the learning ends, agents with the worst performances eliminated while surviving agents produce child agents with random mutation to maintain the size of the population. ES ES-based methods have low sample efficiency since agents the population do not share their experiences and many samples wasted eliminated agents. Fur-
given task. This is very challenging problem for main reasons. First,

---

Published as a conference paper at ICLR 2022
thermore, zeroth-order optimization methods such as ES are known to be sample-inefficient when the optimization search space (i.e., design space) is high-dimensional (Vemula et al., 2019).
In light of the above challenges. take new approach to agent design optimization by incor- porating the design procedure of an agent into its decision-making process. Specifically, we learn transform-and-contee
policy, called Transform2Act, that first designs agent and then controls the designed agent. In an episode, we divide the agent's 's behavior into two consecutive stages, a transform stage and an execution stage, on which the policy is conditioned. In the transform stage, the agent applies a sequence of transform actions to modify its skeletal structure and joint attributes without interacting with the environment. In the execution stage, the agent assumes the design result- ing from the transform stage and applies motor control actions to interact with the environment and receives rewards. Since the policy needs to be used across designs with a variable number of joints, we adopt graph neural networks (GNNs) as the policy's 's main network architecture. Each graph node in the GNNs represents joint and uses message passing with its neighbors to output soint-specific actions. While GNNs can improve the generalizability of learned control policies across different skeletons through weight sharing, they can also limit the specialization of each joint's s design since similar joints tend to output similar transform actions due the weight sharing in GNNs. To tackle this problem, we propose to attach a joint-specialize GNNs in the policy. The JSMLP uses different sets of weights for each joint, which allows more flexible transform actions to enable asymmetric designs with more specialized joint functions.
multilayer perceptron (JSMLP) on top of the
The proposed Transform2Act policy jointly optimizes its transform and control actions via policy gradient methods, where the training batch for each iteration includes samples collected under vari- ous designs. Unlike zeroth-order optimization methods (e.g.. ES) that do not use policy to change designs but instead mutate designs randomly, our approach stores information about the goodness of a design into our transform policy and uses to select designs to be tested in the execution stage. The use of GNNs further allows the information stored in the policy to be shared across joints, enabling better ex experience sharing and prediction generalization for both design and control. Furthermore, in contrast to ES- methods that do not share training samples across designs in generation, our approach uses all the samples from all designs to train our policy, which improves sample efficiency.
The main contributions of this paper are: (1) We propose a mransorm-annd-contrl paradigm that formulates design optimization as learning a conditional policy, which can be solved using the rich tools of RL. (2) Our GNN- -bsed conditional policy enables joint optimization of design and control as well as experience sharing across all designs, which improves sample efficiency substantially. (3) We further enhance the GNNs by proposing a joint-specialized MLP to balance the generalization and specialization abilities of our policy. (4) Experiments show that our approach outperforms previous methods significantly in terms of convergence speed and final performance, and is also able to discover familiar designs similar to giraffes, squids, and spiders.
2 RELATED WORK
Continuous Design Optimization. Considerable research has examined optimizing agent's continuous design parameters without changing its skeletal structure. For instance, Baykal & Al- terovitz (2017) introduce simulated annealing-based optimization framework for designing piece- wise cylindrical robots. Alternatively, trajectory optimization and the implicit function theorem have been used to adapt the design of legged robots (Ha et al., 2017; 2018; Desai al., 2018). Re- cently, deep RL has become a popular approach for design optimization. Chen et al. (2020) model robot hardware as part of the policy using computational graphs. Luck et al. (2020) learn design- conditioned value function and optimize design via CMA-ES. Ha (2019) uses population- policy gradient method for design optimization. Schaff et al. (2019) employs RL and evolutionary strategies to maintain a distribution over design parameters. Another line of work (Yu et al., 2018; Exarchos et al., 2020; Jiang et al., 2021) uses RI. to find the robot parameters that best fit an incom- ing domain. Unlike the above works, our approach can optimize the skeletal structure of an agent in addition to its continuous design parameters.
Combinatorial Design Optimization. Towards jointly optimizing the skeletal structure and design parameters, the seminal work by Sims (1994) uses evolutionary search (ES) to optimize the design and control of 3D blocks. Extending this method, Cheney et al. (2014; 2018) adopt oscillating 3D voxels as building blocks to reduce the search space. Desai et al. (2017) use human-in -annan----the-loop the-loop tree search to optimize the design of modular robots. Wang et al. (2019) propose an evolutionary
2

---

Published as a conference paper at ICLR 2022
graph search method that uses GNNs to enable weight sharing between an agent and its offspring. Recently, Hejna et al. (2021) employ an information cetmmtttoo-thorrticc objective to evolve task task-agnostic agent designs. Due to the combinatorial nature of skeletal structure optimization, most prior works use ES- based optimization frameworks which can be sample-ineffiient since agents with various designs in the population learn independently. In contrast, we learn a mansform-and-connrol policy using samples collected from all designs, which improves sample efficiency significantly.
GNN-based Control. Graph neural networks (GNNs) (Scarselli et al., 2008; Bruna et al., 2013; Kipf & Welling, 2016) are a class of models that use message passing to aggregate and extract features from a graph. GNN-based control policies have been shown to greatl improve the generalizability of learned controllers across agents with different skeletons (Wang et al., 2018; 2019; Huang et al., 2020). Along this line, Pathak et al. (2019) use GNN-based policy to control self-assembling modular robots to perform tasks. Recently, Kurin et al. (2021) show that GNNs can hinder learning in incompatible multitask RI. and propose to use attention mechanisms instead. In this paper, we also study the lack of per-joint specialization caused by GNNs due to weight sharing, and we propose a remedy, joint-specialized MLP, to improve the specialization of GNN-based policies.
BACKGROUND
Reinforcement Learning. Given an agent interacting with an episodic environment, reinforcement learning (RL) formulates the problem as a Markov Decision Process (MDP) defined by tuple M = (S, T. R. 7) of state space, action space, transition dynamics, a reward function, and a discount factor. The agent's 's behavior is controlled by a policy π (a, sz), which models the probability of choosing an action a E given the current state 8 E S. Starting from some initial state 8o, the agent iteratively samples an action from the policy ι and the environment generates the next state S141 based on the transition dynamics T(5c+1| ar) and also assigns a reward r, to the agent. The goal of RL is to learn an optimal policy π² that maximizes the expected total discounted reward received by the agent: (TT = E, E
where H is the variable time horizon. In this paper,
we use a standard policy gradient method, PPO (Schulman et al., 2017), to optimize our policy with both transform and control actions. PPO is particularly suitable for our approach since it has KI. regularization between current and old policies, which can prevent large changes to the transform actions and the resulting design in each optimization step, thus avoiding catastrophic failure.
Design Optimization. An agent's design D E DD plays an important role its functionality. In our setting, the design D includes both the agent's skeletal structure and joint-specific attributes (c.g.. bone length, size, and motor strength). To account for changes in design, we now consider a more general transition dynamics T(s++1 {$t, ae, D) conditioned on design D. The total expected
reward is also now function of design D: [ D) ET,D E r're] One main difficulty of design optimization arises from its bi-level nature, i.e., we need to search over a large design space and solve for the optimal policy under each candidate design for evaluation. Formally, the bi-level optimization is defined as:
D" arg max J)
(1)
D
subject to
TD = arg max J(x, D)
(2)
The inner optimization described by Equation (2) typically requires RL, which is computationally expensive and may take up to several days depending on the task. Additionally, the design space D is extremely large and combinatorial due to a vast number of possible skeletal structures. To tackle these problems, in Section 4 we will introduce a new transform-and-comtrll paradigm that formulates design optimization learning conditional policy to both design and control the agent. uses first-order policy optimization via policy gradient methods and enables experience sharing across designs, which improves sample efficiency significantly.
Graph Neural Networks. Since our goal is to learn a policy to dynamically change an agent's design, we need network architecture that can deal with variable input sizes across different skeletal structures. As skeletons can naturally be represented as graphs, graph neural networks (GNNs) (Scarselli et al., 2008; Bruna et al., 2013; Kipf & Welling, 2016) are ideal for our use case.
We denote a graph as G = EVE,AA where u E V and e E E are nodes and edge respectively, and each node u also includes input feature F., A. A GNN uses multiple GNN layers extract and
3

---

Published as a conference paper at ICLR 2022
aggregate features from the graph G through message passing. For the i-th of N GNN layers, the message passing process can be written as:
m = M(h
(3)
< C(fm) Ve E ((u))).
(4)
hy - Unh ,(),
(5)
where a message sending module M first computes message mi, for each node u from the hidden features hi-1 (h" = σy) of the previous layer. Each node's message is then sent to neighboring nodes Mu), and aggregation module C summarizes the messages received by every node and outputs a combined message c., Finally, a state update module U updates the hidden state h of each node using e¹ and previous hidden states h After N GNN layers, output module P is often used to regress the final hidden features H to desired outputs yo = P(h)). Different designs of modules M, C,C,U, U,, P lead to many variants of GNNs (Wu et al., 2020). We the following equation to summarize GNNs' operations:
yu GNN A; E) where GNN -) is used to denote the output for node u.
4 TRANSFORM2ACT: A TRANSFORM-AND-CONTIR
POLICY
To tackle the challenges in design optimization, our key approach is to incorporate the design proce- dure of an agent into its decision-making process. In each episode, the agent's behavior is separated into two consecutive stages: (1) Transform Stage, where the agent applies transform actions to modify its design, including skeletal structure and joint attributes, without interacting with the en- vironment; (2) Execution Stage, where the agent assumes the new transformed design and applies motor control actions to interact with the environment. In both stages, the agent is governed by a conditional policy, called Trunsform2Aci, that selects transform or control actions depending on which stage the agent in. In the transform stage, no environment reward assigned to the agent, but the agent will see future rewards accrued in the execution stage under the transformed design, which provide learning signals for the transform actions. By training the policy with PPO (Schul- man et al., 2017), the transform and control actions are optimized jointly to improve the agent's s performance for the given task. An overview of our method is provided in Figure 2. We also outline approach in Algorithm 1. In the following, first introduce the preliminaries before describing the details of the proposed Transform2Act policy and the two stages.
Design Representation. To represent various skeletal structures, we denote an agent design graph D. = (V,, Fg,, Ae) where each node u E V2 represents a joint u in the skeleton and each edge E E, represents a bone connecting two joints, and Zu,t E A, is a vector representing the attributes of joint including bone length, size, motor strength, etc. Here, the design D, is indexed by since it can be changed by transform actions during the transform stage.
MDP with Design. To accommodate the transform actions and changes in agent design D,, we redefine the agent's MDP by modifying the state and action space as well as the transition dynamics. Specifically, the new state 8t (sf, Dr, &.) includes the agent's 's state sf in the environment and the agent design D. = (Vi, Fr, Ac), as well as a stage flag +; The new action a E (af, af) consists of both the transform action at and the motor control action of. The new transition dynamics T(s Dr+1, 2 st, Dr, Pe, ae) reflects the changes in the state and action.
Transform2Act Policy. The policy To with parameters 0 is a conditional policy that selects the type of actions based on which stage the agent is in:
T D., 1b) { (af Dr, +2), if t. = Execution
TH Dr 2.), if o2 Transform
where two sub-policies = and тī are used in the transform and execution stages respectively. As the agent in the transform stage does not interact with the environment, the transform sub-policy Th (a.) Dr. 22) is not conditioned on the environment state sf and only outputs transform actions af The execution sub-policy T <a[aass] D., (b) is conditioned on both sf and the design D, to output control actions of. D, is needed since the transformed design will affect the dynamics of the environment in the execution stage. As a notation convention, policies with different superscripts (e.g., m and n do not share the same set of parameters.
4

---

Published as a conference paper at ICLR 2022
4 Skeleton Transform
d Attribute Transform
1 Execution
aeato Poice
Ireators Faiay
Trankk Pudley
Trunk
Cheleton
Simiarion
Sinibiiion
Trandher
Transfum
Transfur
Torefom
Legends
NoCrange
d
Figure 2: Transform2Act divides episode into three stages: (1) Skeleton transform stage, where sub-policy ze changes the agent's skeleton by adding or removing joints; (2) Autribute transform stage, where sub-policy = changes joint attributes (e.g., length, size); (3) Execution stage, sub- policy gH selects control actions for the newly-designed agent to interact with the environment.
4.1 TRANSFORM STAGE
In the transform stage, starting from initial design Do. the agent follows the transform sub- policy Ty (at) D., (2) which outputs transform actions to modify the design. Since the design D. = (V,, Ee, A,) includes both the skeletal graph (V,, F2,) and joint attributes A,, the transform action af E {aj, az} consists of two types: (1) Skeleton Transform Action a², which is discrete and changes the skeletal graph (V,, Ee) by adding deleting joints; (2) Attribute Transform Action of which modifies the attributes of each joint and can be either continuous or discrete. Based on the two types of transform actions, we further divide the transform stage into two sub-stages Skeleton Transform Stage and Attribute Transform Stage where ot and af are taken by the agent respectively. We can then write the transform sub-policy as conditioned on the agent's 's stage:
( D2), if 4; = Skeleton Transform
T (a 21)
(8)
(ā Dr, Øt), if d; Attribute Transform
where the agent follows sub-policies т for N₂ timesteps in the skeleton transform stage and then follows ná for N₂ timesteps in the attribute transform stage. It may be tempting merge the two stages and apply skeleton and attribute transform actions together. However, we separate the two stages since the skeleton action increase the number of joints in the graph, while attribute trans- form actions can only output attribute changes for existing joints.
Skeleton Transform. The skeleton transform sub-policy (g(aj] tt) adopts GNN-based net- work, where the action of {a E Vi} is factored across joints and each joint u outputs its categorical action distribution म ( Dr, &.). The policy distribution the product of all joints' action distributions:
( (D. tb)) = Clai, lu,t GNN,(u, AG; Vz, Ed). Vu E Ve.
(9)
< Dr, 21) = II Tā(a", (a 4,),
(10)
MEV,
where the GNN uses the joint attributes A, as input node features to output the logits Lu,s of each joint's categorical action distribution C. The skeleton transform action of. has three choices:
AddJoint joint u will add a child joint v to the skeletal graph, which inherits its attribute Zu,t-
DelJoint: joint u will remove itself from the skeletal graph. The action is only performed when the joint u has no child joints, which is to prevent aggressive design changes.
NoChange: no changes will be made to joint u.
After the agent applies the skeleton transform action d at each joint u, we obtain the design D++1 = (V++1, E,41, At+1) for the next timestep with a new skeletal structure.
The GNN-based skeleton transform policy enables rapid growing of skeleton structures since every joint can add a child joint at each timestep. Additionally, also encourages symmetric structures due to weight sharing in GNNs where mirroring joints choose the same action.
5

---

Published as a conference paper at ICLR 2022
Algorithm 1: Agent Design Optimization with Transform2Act Policy
initialize Transform2Act policy Te
while not reaching max iterations do
memory M t 0;
while M not reaching batch size do
Do < initial agent design;
for 0, .,.,,...4 N2. do
Skeleton Trans form Stage
sample skeleton transform action aj ~Tg: 4 f- Skeleton Transform;
D++1 6- apply oj to modify skeleton (V,, E,) D.:
0; store (ri, o;; D,, 4,) into MM;
for t = , .Na N2 do
// Attribute Transform Stage
sample attribute transform action az T.. 4 t- Attribute Transform; D++1 f- apply at to modify attributes A, in D;;
Tr + 0; store (rr, at, D,, 2r) into M:
NN+NN, <- initial environment state:
for t N₂ + -N H do
// Exccution stage
sample motor control action of Tg: +, t Exceution; + <- environment dynamics T ( Tt environment reward; store (re, (e, s9, D2s b,) into M
D++1 f- De;
update with PPO using samples in M
// Policy Update
return no
Attribute Transform. The attribute transform sub-policy [T 4,) adopts the same GNN- based network as the skeleton transform sub-policy zß. The main difference that the output action distribution can be either continuous or discrete. In this paper, we only consider continuous attributes including bone length, size, and motor strength, but our method by design can generalize discrete attributes such as geometry types. The policy distribution is defined as:
TH T P.) N( p E²),
F GNN, A V., E.), Vu Vz, II Tā(a", Dr, 2b). VeV,
(11)
(a 2.)
(12)
where the GNN outputs the mean ph of joint u's Gaussian action distribution and Σn a learnable diagonal covariance matrix independent of Dr, 2, and shared by all joints. Each joint's action a²,t is used to modify its attribute feature: 2,,411 = Za,t + a and the new design becomes Det1 = (V,, E,, As41) where the skeleton (V, E,) remains unchanged.
Reward. During the transform stage, the agent does not interact with the environment, because changing the agent's design such as adding or removing joints while interacting with the environ- ment does not obey the laws of physics and may be exploited by the agent. Since there is no interaction, we do not assign any environment rewards to the agent. While is possible to add re- wards based on the current design to guide the transform actions, in this paper, we do not use any design-related rewards for fair comparison with the baselines. Thus, no reward is assigned to the agent the transform stage, and the transform sub-policies are only trained using future rewards from the execution stage.
Inference. At test time, the most likely action will be chosen by both the skeleton and attribute transform policies. The design D, after the transform stage is the final design output.
4.2 EXECUTION STAGE
After the agent performs N₂ skeleton transform and N, attribute transform actions, enters the execution stage where the agent assumes the transformed design and interacts with the environment. GNN-based execution policy <af]ss D,, 4 is used in this stage to output motor control actions interacts with the environment. the policy лद is conditioned
af for each joint. Since the agent on the environment state sf as well as the transformed design D,, which affects the dynamics of the environment. Without loss of generality, assume the control actions continuous. The

---

Published as a conference paper at ICLR 2022
execution policy distribution is defined as: 166,,, D,, Φ1) (a Pa Ed), Pu,: (u, 66 AG Vo, Ee), Vu Ve: II
(13)
Tg(a Dj, Dt)
T D,, 1t). NEV,
(14)
where the environment state $ = [ Vi} includes the state of each node (e.g.. joint angle and velocity). The GNN uses the environment state &e and joint attributes A as input node features to output the mean ph,,, of each joint's s Gaussian action distribution. Σ is a state-independent learn- able diagonal covariance matrix shared by all joints. The agent applies the motor control actions a all joints and the environment transitions the agent to the next environment state si+1 accord- ing to the environment's 's transition dynamics (5 (s, af). The design D, remains unchanged throughout the execution stage.
4.3 VALUE ESTIMATION
As actor-critic method (PPO) for policy optimization, need to approximate the value function i.c., the expected total discounted rewards starting from state s = ($) D,, 2,):
Vis De, 21) & Eso
Σ Cr're
(15)
We learn a GNN-based value network 1/s with parameters & to approximate the true value function:
(s D,, p.) = GNN, st, Ac; V,, Ex)
(16)
where the GNN takes the environment state st. joint attributes A, and stage flag t (one-hot vector) as input node features to output scalar at each joint. We use the output of the root joint as the predicted value. The value network Vos is used in all stages. In the transform stage, the environment state 8 is unavailable so we set it to O.
4.4 IMPROVE SPECIALIZATION WITH JOINT- MLP
As demonstrated in prior work (Wang et al., 2018; Huang et al., 2020), GNN-based control poli- cies enjoy superb generalizability across different designs. The generalizability can be attributed to GNNs' weight sharing across joints, which allows new joints to share the knowledge learned by existing joints. However, weight sharing also means that joints in similar states will choose similar actions, which can seriously limit the transform policies and the per-joint specialization of the re- sulting design. Specifically, as both skeleton and attribute transform policies are GNN-based, due to weight sharing, joints in similar positions in the graph will choose similar or the same transform ac- tions. While this does encourage the emergence of symmetric structures, it also limits the possibility of asymmetric designs such as different lengths, sizes, or strengths of the front and back legs.
To improve the per-joint specialization of GNN-based policies, we propose to add a joint-specialized multi-layer perceptron (JSMLP) after the GNNs. Concretely, the JSMLP uses different MLP weights for each joint, which allows the policy to output more specialized joint actions. To achieve this, we design a joint indexing scheme that is consistent across all designs and maintain joint-indexed weight memory. The joint indexing is used identify joint correspondence across designs where two joints with the same index are deemed the same. This allows the same joint in different designs to use the same MIP weights. However, we cannot simply index joints using a breadth-firs search (BFS) for each design since some joints may appear or disappear across designs, and completely different joints can be assigned the same index in BFS. Instead, we index a joint based on the path from the root the joint. The details of the indexing provided in Appendix C. As we will show in the ablation studies, JSMLPs can improve the performance of both transform and control policies.
5 EXPERIMENTS
We design our experiments to answer the following questions: (1) Does our method, Transform2Act, outperform previous methods in terms of convergence speed and final performance? (2) Does Transform2Act create agents that look plausible? (3) How do critical components - GNNs and JSMIPP- affect the performance of Transform2Act?
7

---

Published as a conference paper at ICLR 2022
20 Locomotion
3D Locomotion
Swimmer
Crosser
uertomaacc swrt
grwntn m
Powardd
Roward
Heward
Sinulation Step (Millio)
Sinulaion Step (Millio)
Stimulation Step (Million)
Simulation Step (Millio)
nbsl Design
Istal Desigr
Inttall Design
Inita Desig
NCF
NSF
Ner
NGF
R25
RG3
RGS
RG5
Taraform22ct Duun)
TranfomiAes
TranafommAct (uus)
TranffommAe Curs
Dung
Figure 3: Baseline comparison. For each environment, we plot the mean and standard deviation of total rewards against the number of simulation steps for all methods, and show their final designs.
Environments. We evaluate Transform2Act four distinct environments using the MuJoCo sim- ulator (Todorov et al., 2012): (1) 2D Locomotion, where 2D agent living in an z:z-laae is tasked with moving forward fast possible, and the reward is its forward speed. (2) 3D Locomotion, where a 3D agent's ss goal is to move as fast as possible along æ-axis and is rewarded by its forward speed along r-axis. (3) Swimmer, where a 2D agent living water with 0.1 viscosity and confined in an zy-plane is rewarded by its moving speed along r-axis. (4) Gap Crosser, where a 2D agent living in an x2 -la-p needs cross periodic gaps and rewarded by forward speed.
Baselines. We compare Transform2Act with the following baselines that also optimize both the skeletal structure and joint attributes of agent: (1) Neural Graph Evolution (NGE) (Wang al., 2019), which is an ES-based method that uses GNNs to enable weight sharing between agent and its offspring. (2) Evolutionary Structure Search (ESS) (Sims, 1994), which classic ES- method that has been used in recent works (Cheney et al., 20 14; 2018). (3) Random Graph Search (RGS), which baseline employed in (Wang et al., 2019) that trains population of agents with randomly generated skeletal structures and joint attributes.
5.1 COMPARISON WITH BASELINES
In Figure we show the learning curves of each method and their final agent designs for all four environments. For each method, the learning curve plots use six seeds per environment and plot the agent's 's total rewards against the total number of simulation steps used by the method. For ES-based baselines with population of agents, we plot the performance of the best agent at each iteration. We can clearly see that our method, Transform2Act, consistently and significantly outperforms the baselines in terms of convergence speed and final performance.
Next, let us compare the final designs generated by each method in Figure 3. For better comparison, encourage the reader see these designs in video on the project website. For 2D locomotion, Transform2Act is able to discover giraffe-like agent that can run extremely fast and remain stable. The design is suitable for optimizing the given reward function since it has long neck to increase its forward momentum and balance itself when jumping forward. For 3D Locomotion, Transform2Act creates spider-like agent with long legs. The design is roughly symmetric due to the GNN-based
8

---

Published as a conference paper at ICLR 2022
Locomotion
3D Locomotion
Swimmer
Gap Crosser
Sumber del Chec
Roward
Reward
Reward
Evm mo
Rin
220
2009
13
20
Simdiatinn Step (Millio)
Simulation Step (Millon)
Simulation Step (Millon)
Simuiation Step (ASlloo))
Figure 4: Ablation studies. The plots indicate that GNNs and JSMLPs both contribute to the
performance and stability our approach greatly.
Locomotion
Swimmer
Ours
Ours wio All JSMIPPs
Ours
Ours w/o All JSMLPs
Reward:
Figure 5: Effect of JSMLPs. Designs without JSMLPs are overly symmetric with little per-joint
specialization, which leads to worse performance.
transform policy, but it also contains joint-specific features thanks to the JSMLPs which help the agent attain better performance. For Swimmer, Transform2Act produces squid-like agent with long bodies and numerous tentacles. As shown in the video, the movement of these tentacles propels the agent forward swiftly in the water. Finally, for Gap Crosser, Transform2 designs a Hopper- like agent that can jump across gaps. Overall, we can see that Transform2Act is able to find plausible designs similar to giraffes, squids, and spiders, while the baselines fail to discover such designs and have much lower performance in all four environments. We also provide detailed discussion of our method's advantages over the strong ES-based baseline, NGE (Wang et al., 2019), in Appendix B.
Continuous Design Optimization. In some cases, we may already have good agent designed by experts and want to keep its skeletal structure. In such scenarios, we can also apply Transform2Act to finetune the expert design's attributes to further improve its performance. We show that our method outperforms a popular continuous design optimization approach (Ha, 2019) in Appendix A.
5.2 ABLATION STUDIES
We aim to investigate the importance of two critical components in our approach - GNNs and JSMLPs. We design three variants of our approach: (1) Ours w/o GNNs, where remove all the GNNs from our Transform2Act policy and uses JSMLP only; (2) Ours w/o Control JSMLP. where we remove the JSMLP from our execution sub-policy Tß: (3) Ours w/o All JSMLPs, where we remove all the JSMLPs from the execution sub-policy TH and transform sub-policies TÔ and Th- The learning curves of all variants are shown in Figure It evident that GNNs are crucial part of our approach as indicated by the large decrease in performance of the corresponding variant. Ad- ditionally, JSMLPs are very instrumental for both design and control in our method, which not only increase the agent's 's performance but also make the learning more stable. For Swimmer, the variant without JSMLPs has a very large performance variance. To further study the effect of JSMLPs, we also show the design with and without JSMLPs for 3D Locomotion and Swimmer in Figure 5. We can observe that the designs without JSMLPs are overly symmetric and uniform while the designs with JSMLPs contain jpent-specialized features that help the agent achieve better performance.
CONCLUSION
In this paper, we proposed new cansform-and-comee paradigm that formulates design optimiza- tion conditional policy learning with policy gradients. Compared to prior ES-based methods that randomly mutate designs, our approach is more sample-efficient as it leverages a parameterized pol- icy to select designs based on past experience and also allows experience sharing across different designs. Experiments show that our approach outperforms prior methods in both convergence speed and final performance by order of magnitude. Our approach also automatically discover plau- sible designs similar to giraffes, squids, and spiders. For future work, we are interested in leveraging RI. exploration methods to further improve the sample efficiency of approach.

---

Published as a conference paper at ICLR 2022
Acknowledgements. This work is supported by the NVIDIA Graduate Fellowship.
REFERENCES
Cenk Baykal and Ron Alterovitz. Asymptotically optimal design of piecewise cylindrical robots using motion planning. In Robotics: Science and Systems, volume 2017, 2017.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint /rXiv: 2016.
Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs. arXiv preprint urXivv/312..60 2013.
Tianjian Chen, Zhanpeng He, and Matei Ciocarlie. Hardware as policy: Mechanical and compu- tational co-optimization using deep reinforcement learning. arXiv preprint arXiv:2008. 04460, 2020.
Nick Cheney, Robert MacCurdy, Jeff Clune, and Hod Lipson. Unshackling evolution: evolving soft robots with multiple materials and powerful generative encoding. ACM SIGEVOlution, 7(1): 11-23, 2014.
Nick Cheney, Josh Bongard, Vytas SunSpiral, and Hod Lipson. Scalable co-optimization of mor- phology and control in embodied machines. Journal of The Royal Society Interface, 15(143): 20170937, 2018.
Ruta Desai, Ye Yuan, and Stelian Coros. Computational abstractions for interactive design of robotic devices. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 1196- 1203. IEEE, 2017.
Ruta Desai, Beichen Li, Ye Yuan, and Stelian Coros. Interactive co-design form and function for legged robots using the adjoint method. arXiv preprint arXiv:1801.00385, 2018.
Ioannis Exarchos, Yifeng Jiang, Wenhao Yu, and C Karen Liu. Policy transfer via kinematic domain randomization and adaptation. arXiv preprint arXiv:201 1,01891, 2020.
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. 2019.
arXiv preprint
David Ha. Reinforcement learning for improving agent design. Artificial life, 25(4):352-365, 2019.
Schoon Ha, Stelian Coros, Alexander Alspach, Joohyung Kim, and Katsu Yamane. Joint optimiza- tion of robot design and motion parameters using the implicit function theorem. In Robotics: Science and systems, volume 8, 2017.
Sehoon Ha, Stelian Coros, Alexander Alspach, Joohyung Kim, and Katsu Yamane. Computational co-optimization of design parameters and motion trajectories for robotic systems. The Interna- tional Journal of Robotics Research, 37(13-14)15211-533 2018.
Donald Hejna, Pieter Abbeel, and Lerrel Pinto. Task-agnostic morphology evolution. arXiv preprint 1:Xi::2102.13100 2021.
Wenlong Huang, Igor Mordatch, and Deepak Pathak. One policy control them all: Shared modular policies for agnnsstg control. In International Conference on Machine Learning, pp. 4455- 4464. PMLR. 2020.
Yifeng Jiang, Tingnan Zhang, Daniel Ho, Yunfei Bai, C Karen Liu, Sergey Levine, and Jie Tan. Sim- gan: Hybrid simulator identification for domain adaptation via adversarial reinforcement learning. arXiv preprint ar arXiv:2101.06005 2021.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net-
works. arXiv preprint arXiv: 1609.02077 2016.
10

---

Published as a conference paper at ICLR 2022
Vitaly Kurin, Maximilian Igl, Tim Rocktäschel, Wendelin Boehmer, and Shimon Whiteson. My body is a cage: the role of morphology in graph-based incompatible control. In International Conference Learning Representations, 2021. URL htRp:///s///peeee http:///ee//peeee...eeo.. forum?
id=N32UDGN510,
Kevin Sebastian Luck, Heni Ben Amor, and Roberto Calandra. Data-efficient co-adaptation of morphology and behaviour with deep reinforcement learning. Conference on Robot Learning, pp. $69. PMLR, 2020.
Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 4602-4609) 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32: 8026-8037, 2019.
Deepak Pathak, Chris Lu, Trevor Darrell, Phillip Isola, and Alexei A Efros. Learning to con- trol self-assembling morphologies: study of generalization via modularity. arXiv preprint 1rXiv:1902.05546, 2019.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini.
The graph neural network model. IEEE transactions on neural networks, 20(1):61-80, 2008
Charles Schaff, David Yunis, Ayan Chakrabarti, and Matthew R Walter. Jointly learning con- struct and control agents using deep reinforcement learning. In 20/9 International Conference on Robotics and Automation (ICRA), pp. 9798-9805. IEEE, 2019.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Picter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint 1:Xiv:506.002430 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
Karl Sims. Evolving virtual creatures. In Proceedings of the 21st annual conference on Computer graphics and interactive techniques, pp. 15-22, 1994.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033 IEEE, 2012.
Anirudh Vemula, Wen Sun, and Bagnell. Contrasting exploration in parameter and action space: A zeroth-order optimization perspective. In The 22nd International Conference on Artificial In- telligence and Statistics, pp. 2926-2935. PMLR, 2019.
Tingwu Wang, Renjie Liao, Jimmy Ba, and Sanja Fidler. Nervenet: Learning structured policy with graph neural networks. In International Conference on Learning Representations, 2018.
Tingwu Wang. Yuhao Zhou, Sanja Fidler, and Jimmy Ba. Neural graph evolution: Towards efficient automatic robot design. arXiv preprint arXiv: 196.v0190.005370, 2019.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 32(1))4-24, 2020.
Wenhao Yu, C Karen Liu, and Greg Turk. Policy transfer with strategy optimization. arXiv preprint arXiv: 181000551, 2018
11

---

Published as a conference paper at ICLR 2022
A COMPARISON WITH CONTINUOUS DESIGN OPTIMIZATION BASELINES
2D Locomotion
3D Locomotion
6000
Transferrldd
Expert
Cesign
for fted on (it 2019)
A1. bedign b-a 20191
6000
Reward
Reward
4000
5,00
3000
4.00
3000
2000
1000
1000
10
20
30
40
0
10
20
30
40
50
Simulation Step (Million)
Simulation Step (Million)
Swimmer
Gap Crosser
Fxpert
5000
teeiin
800
NL Dica 2015)
BL Design 2013)
Reward
Reward
600
400
200
1900
10
20 30 40
50
10
20
3G
40
50
Simulation Step (Million)
Simulation Step (Million)
Figure 6: Baseline comparison of continuous design optimization for finetuning expert designs. The expert designs are taken from OpenAl Gym. where we use Hopper for 2D Locomotion and Gap Crosser, Ant for 3D Locomotion, and Gym Swimmer for our Swimmer.
In this section, we aim to evaluate our approach's ability to finetune a given expert design to further improve its performance. To this end, we perform experiments with the skeleton transform stage disabled in our approach and use the expert design as the initial design. The expert designs taken from OpenAl Gym (Brockman et al., 2016), where we use Hopper for 2D Locomotion and Gap Crosser, Ant for 3D Locomotion, and Gym Swimmer for our Swimmer. We compare our method against a popular RL-based continuous design optimization approach (Ha, 2019). We use the same network architectures for all methods to ensure fair comparison. The policy for the expert design is trained with PPO (Schulman et al., 2017). The results are shown in Figure 6 where we use six seeds per environment for each method. We can see that our approach converges much faster than the expert design and baseline (Ha, 2019) for all environments. Our method also achieves significantly better final performance after 50 million steps of training. This demonstrates that, in addition to combinatorial design optimizatin, our approach also enjoys superior performance in continuous design optimization, which has many real-world applications.
B ADVANTAGES OF TRANSFORM2ACT OVER NGE
There are three main advantages of our method over NGE (Wang et al., 2019):
1. NGE does not allow experience sharing among species in a generation. As shown in Algo- rithm of NGE, each species inside a generation j has its own set of weights 0f and is trained independently without sharing experiences among different species. The experience sharing in NGE is only enabled through weighting sharing between a species and its parent species from the previous generation. This means that if there are N species in a generation, in every epoch, each species is only trained with M/N number of experience samples where M is the sample budget for each epoch. At the end of the training, each species has only used EM/ /M/N samples for training where E is the number of epochs. In contrast, in our method, every design shares the same control policy, so the policy can use all EM samples. Therefore, our method allows better experience sharing across different designs, which improves sample efficiency.
12

---

Published as a conference paper at ICLR 2022
2. Our method uses transform policy to change designs instead of random mutation. Our transform policy takes the current design as input to output the transform actions (design changes). Through training, the policy learns to store information about which design to prefer and which design to avoid. This information is also shared among different joints via the use of GNNs, where joints in similar states choose similar transform actions (which is also bal- anced by JSMLPs for joint specialization). Additionally, the policy also allows every joint to simultaneously change its design (e.g., add child joint, change joint attributes). For example, in 3D Locomotion, the agent can simultaneously grow its four feet in single timestep, while ES-based methods such as NGE will take four different mutations to obtain four feet. There- fore, our method with the transform policy allows better generalization and experience sharing among joints, compared to ES- ES-based methods that perform random mutation.
3. Our method allows more exploration. Our transform-and-control policy tries out new de- sign every episode, which means our policy can try M // designs every epoch, where M is the total number of sample timesteps and Havg is the average episode length. There is also more exploration for our approach at the start of the training when Havg is small. On the other hand, ES- methods such as NGE only try N (num of species) different designs every epoch. If NGE uses too many species (large N), each species will have few samples to train as mentioned in point 1. Therefore, N is typically set to be < Hwg. For example, N is set from 16 to 100 in NGE, while M/ Hagg in method can be more than 2000.
C DETAILS OF
MLP (JSMLP)
111
11
11
31
21
211
31
111
211
Agent Design
Skeletal Tree
Rearranged Skeletal Tree
Figure 7: An example of the joint indexing used by the JSMLPs in approach. Every non-root joint is indexed by the path from the root to the joint. Starting from an empty index string at the root, every time we go the i-th child of the current joint, we add i to the left side of the index string. For example, the joint '211' gets its index because to reach the joint from the root, we need to first go the first child 'I' of the root, and then go to the first child '11' of joint '1', and then go to the second child '211' I' of joint "11', which is the target joint.
As discussed in Section 4.4, JSMLPs use different sets of MLP weights for each joint improve the per-joint specialization ability of our GNN-based policy networks. To achieve this, we need a joint indexing retrieve MLP weights from the joint-indexed weights memory. The joint indexing is used to identify joint correspondence across designs where two joints with the same index are deemed the same. This allows the same joint in different designs to use the same MI.P weights.
Figure illustrates our joint indexing method. We first assign '0' to the root joint. For any non-root joint, its index is based on the path from the root the joint. Starting from an empty string at the root, we follow the path to the target joint (to be indexed) down the skeletal tree, and every time we go to the i-th child of the current joint, we add to the left side of the index string. For example, to reach the rightmost joint from the root, adds 'I' to the empty string and obtains '1'; we then need to go to the third child of the current joint '1', which adds '3' the current string '1' and obtains '31', which is the final index string for the rightmost joint since we have reached it. When a joint is removed and the design is changed, we
first need to go to the first child of the root, which
reindex the joints based on the updated skeletal graph. We can also very easily convert the index string into an integer using base No + 1 where No is the maximum number of children each joint allowed have.
13

---

Published as a conference paper at ICLR 2022
D VISUALIZATIONS OF TRANSFORM ACTIONS
Figure 8: Designs process of Transform2Acc for the 2D Locomotion environment. We visualize the design after each skeleton transform action. For better visualization, the joint attributes after the attribute transform stage.
t=0
t=1
Figure 9: Designs process of Transform2Act for the 3D Locomotion environment. We visualize the design after each skeleton transform action. For better visualization, the joint attributes after the attribute transform stage.
t=0
t=1
t=2
t=3
t=4
t=5
Figure 10: Designs process of Transform2Act for the Swimmer environment. We visualize the design after each skeleton transform action. For better visualization, we use the joint attributes after the attribute transform stage.
14

---

Published as a conference paper at ICLR 2022
E ENVIRONMENT DETAILS
2D Locomation
3D Locomation
Swimmer
Gap Crosser
Figure 11: A random agent in each of the four environments.
In this section, provide additional details about the four environments use in our experiments. We show what a random agent looks like in each environment in Figure 11. Each joint the agent uses hinge connection with its parent joint. Each joint's environment state sy, includes its joint angle and velocity. For the root joint, we also include other information such as the height, phase and world velocity. Zero padding used to ensure that each joint's state has the same length. Each joint's 's attribute feature Zu,t includes the bone vector, bone size, and motor gear value of the joint. Each attribute normalized within the range [-1, 1] using loosely defined attribute range.
2D Locomotion. The agent in this environment lives inside an zz-plane with flat ground at = = 0. Each joint of the agent allowed to have maximum of three child joints. For the root joint, we add its height and 2D world velocity to the environment state. The reward function is defined as
= Ff p. / 1
(17)
where of denotes the z-position of the agent and öt = 0.008 is the time step. An alive bonus of is also used inside the reward. The episode is terminated when the root height is below 0.7.
3D Locomotion. The agent in this environment lives freely in a 3D world with a flat ground at O. Each joint of the agent is allowed to have maximum of two child joints except for the root. For the root joint, we add its height and 3D world velocity to the environment state. The reward function is defined
1 Σ
Tt P.+1
p
/8t w
I|
(18)
WEV
where 0.0001 is weighting factor for the control penalty term, J is the total number of joints, and the time step is 0.04.
Swimmer. The agent in this environment lives in water with 0.1 viscosity and is confined within an ry-plane contacting a flat ground. Each joint of the agent is allowed to have a maximum of three child joints. For the root joint, add its 2D world velocity the environment state. The reward function is the same as the one used by 3D Locomotion as defined in Equation (18).
Gap Crosser. The agent in this environment lives inside xz=plane. Unlike other environments, the terrain of this environment includes a periodic gap of 0.96 width with the period being 3.2. The height of the terrain is 0.5. The agent must cross these gaps to move forward. Each joint of the agent is allowed to have maximum of three child joints. For the root joint, we add its height, 2D world velocity, and phase variable encoding the periodic z-position of the agent the environment state. The reward function is defined as
= Pi+1 - p. /5t + 0.1,
(19)
where the time step öt is = 0.008. An alive bonus of 0.1 is also used inside the reward. The episode terminated when the root height is below 1.0.
Other Details. Since MuJoCo agents are specified using XML strings, during the transform stage, represent each design an XML string and modify its content based on the transform actions. At the start of the execution stage, the modified XML string is used to reset the MuJoCo simulator and load the newly-designed agent.
15

---

Published as a conference paper at ICLR 2022
F COMPUTATION COST
Following standard RL practices, we use distributed trajectory sampling with mulitple CPU threads accelerate training. For all the environments used in the paper, it takes around one day to train our model on a standard server with 20 CPU cores and an NVIDIA RTX 2080 Ti GPU.
G HYPERPARAMETERS AND TRAINING PROCEDURES
In this section, we present the hyperparameters searched and used for our method in Table and the hyperparameters for the baselines in Table 2. We use Py PyTorch (Paszke al., 2019) to implement all the models. For the implementation of GNNs, we employ the PyTorch Geometric package (Fey & Lenssen, 2019) and use GraphCony (Morris et al., 2019) as the GNN layer. When training the policy with PPO, we also adopt generalized advantage estimation (GAE) (Schulman et al., 2015).
For the baselines, we implement our own versions of NGE, RGS, and ESS following the publicly released code' of NGE (Wang et al., 2019). For NGE, we use the same GNN architecture as the one used our method to make fair comparison. We also ensure that baselines and our method use the same number of simulation steps for optimization. For instance, our method optimizes the policy with batch size 50000 for 1000 epochs, which amounts to 50 million simulation steps. The basclines population of 20 agents, and each is trained with batch size of 20000 for 125 generations, which also amounts to 50 million simulation steps.
Hyperparameter
Values Searched & Selected
Num. of Skeleton Transforms N₂
3, 5, 10
Num. of Attribute Transforms N₂
1, 3, 5
Policy GNN Layer Type
GraphConv Tanh
JSMLP Activation Function
GNN Size (Skeleton Transform)
(32, 32, 32), (64, 64, 64), (128, 128, 128), (256, 256, 256)
JSMLP Size (Skeleton Transform)
(128, 128), (256, 256), (512, 256), (256, 256, 256)
GNN Size (Attribute Transform)
(32, 32, 32), (64, 64, 64), (128, 128, 128), (256, 256, 256)
JSMLP Size (Attribute Transform)
(128, 128), (256. 256), 256), (256, 256, 256)
GNN Size (Execution)
(32, 32, 32), (64, 64, 64), (128, 128. 128), (256, 256, 256)
JSMLP Size (Execution)
(128, 128), (256, 256), (512, 256), (256, 256, 256)
Diagonal Values of Σ
1.0, 0.04, 0.01
Diagonal Values of E
1.0. 0.04, 0.01
Policy Learning Rate
5e-5, le-4, 3e-4
Value GNN Layer Type
GraphConv Tanh
Value Activation Function
Value GNN Size
(32, 32, 32), (64, 64, 64), (128, 128, 128), (256, 256, 256) (128, 128), (256, 256), (512, 256). (256, 256, 256)
Value MLP Size
Value Learning Rate
le-4, 3e-4
PPO clip é
0.2
PPO Batch Size
10000, 20000, 50000 512, 2048
PPO Minibatch Size
Num. of PPO Iterations Per Batch
1. 10
Num. of Training Epochs
1000
Discount factor 7 GAE
0.99, 0.995, 0.997, 0.999 0.95
Table 1: Hyperparameters searched and used by our method. The bold numbers among multiple values are the final selected ones. For Gap Crosser, we use 0.999 for the discount factor 7.
com/ http://wwww..o.oooooooo/////oooo on
16

---

Published as a conference paper at ICLR 2022
Hyperparameter
Values Searched Selected
Num. of Generations
125 10, 20, 50, 100 0.15, 0.2, 0.3, 0.4 GraphCony Tanh
Agent Population Size
Elimination Rate
GNN Layer Type
MLP Activation
Policy GNN Size
(32, 32, 32). (64, 64, 64), (128, 128, 128) (128, 128), (256, 256), (512, 256)
Policy MI.P Size
Policy Log Standard Deviation
0.0, -1.6, -2.3 5e-5, lc-4, 3e-4
Policy Learning Rate
Value GNN Size
(32, 32, 32). (64, 64, 64), (128, 128, 128) (128, 128), (256, 256), (512, 256)
Value MLP Size
Value Learning Rate
le-4, 3e-4
PPO clip
0.2 10000, 20000, 50000 512, 2048
PPO Batch Size
PPO Minibatch Size
Num. of PPO Iterations Per Batch Discount factor ?
1, 5, 10 0.99, 0.995, 0.999 0.95
GAE À
Table 2: Hyperparameters searched and used by the baselines. The bold numbers among multiple values are the final selected ones. For Gap Crosser, we use 0.999 for the discount factor 7.
17


Reviews:
[{"review_id": "iXBSfM8lko", "reviewer": "Reviewer_ZAQQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: clear idea, described well, convincing experiments\n\nWeaknesses:\n1) Formulations of some claims could be made more neutral. E.g., \"improves sample efficiency tremendously\" -> substantially, considerably, significantly\n2) The paper refers to policy gradients (PG) as \"first-order\" optimization methods. This may not be exactly technically correct as these methods still only use zero order information about the objective function. The authors are encouraged to consult the recent literature on the analysis of PG methods, e.g., [1].\n3) Table 1 in Appendix E shows that N_s=5 skeleton transforms and N_z=1 attribute transform were selected using hyperparameter search. What qualitative results does one obtain if there are more than 1 attribute transform? Why allowing more skeleton transforms performs worse? In principle, it should provide more flexibility. Adding a discussion explaining the effects of these numbers to the main body of the paper would be illuminating.\n4) Table 1 in Appendix E shows values -2.3 and 0.0 are selected for diagonal values of covariance matrices Sigma^z and Sigma^e. This seems to be a mistake, because diagonal values should be greater than zero.\n\n[1] Agarwal, A., Henaff, M., Kakade, S., & Sun, W. (2020). Pc-pg: Policy cover directed exploration for provable policy gradient learning. arXiv preprint arXiv:2007.08459.\n\n", "summary_of_the_paper": "The paper proposes an algorithm for simultaneous agent design and policy optimization. The choice of the body structure is treated as another action available to the agent. Therefore, the policy is parameterized by graph neural networks (GNNs), and it outputs i) the skeleton structure, ii) node attributes such as bone length, size, motor strength, and iii) motor control commands. Thanks to the parameterization via GNNs, the policy can be trained with PPO. Experiments show that the proposed method outperforms prior approaches, which mainly employ evolutionary methods for optimization, whereas the proposed method leverages more sample efficient policy gradient algorithms.", "summary_of_the_review": "Strong paper. The proposed approach is novel and can be impactful in other discrete-continuous optimization domains. Experiments validate the advantages of the proposed method. The paper is clear and easy to follow.\n\n====\n\nI am satisfied with the response of the authors and I maintain my score \"8: accept, good paper\".", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "N8Z2nXZKFSD", "reviewer": "Reviewer_mvHK", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Formulating design and control co-optimization as one sequential decision-making problem is novel.\n- The paper's ideas all make sense (transform-and-control policy, skeleton and attribute transform, JSMLP).\n- The empirical results look stronger than existing baselines.\n- The paper is written to be easy to understand.\n\nWeaknesses:\n- Experiments are only conducted on four custom environments. Why not use existing environments from NGE or [1] (see references below)? Also, three random seeds are far below the standard. \n- Little analysis on the empirical results, given no theoretical justification of the algorithm. The analysis can be further enhanced from several aspects: 1) Discussing comparison with NGE (probably the strongest baseline) about similarities and differences and how those differences lead to a huge performance increase; 2) Enriching ablation studies by training with skeleton transformation disabled and attribute transformation disabled respectively.\n\nConcerns:\n- I can't entirely agree with the argument that this approach enables first-order optimization of agent design. Technically, both this approach and ES-based methods do not have access to the ground-truth gradient and estimate first-order gradients based on the gathered experiences. So, this approach is still zeroth-order optimization, and it's not appropriate to claim that the sample efficiency comes from the first-order nature of the method.\n- While I agree that ES-based methods have a high-dimensional search space for design, your approach does not essentially reduce that search space. Instead, the search space for policy is much larger in your formulation, i.e., the dimension of MDP becomes much higher.\n- Could you provide more intuition on the comparison against NGE? By looking at the performance curves, it outperforms NGE by a large margin even without all JSMLPs. Is it because of the new formulation of the design optimization, or the attribute transform is optimized (unlike sampling from uniform distributions in NGE), or other reasons?\n- I'm skeptical about the reported performance of NGE because of several reasons: 1) In the original NGE's paper, it outperforms RGS by a large margin, while in Figure 3 of this paper, the improvement is marginal; 2) I believe NGE should be much better than RGS is because NGE also uses GNN policies that allow experience sharing across different designs; 3) If experience sharing via GNN is not effective in NGE, why this is effective in your approach as you claimed? A good way to address my concerns is to probably run NGE's original implementation besides your own implementation and report the performance.\n\nOther suggestions:\n- Section 4 can be shortened to include more analysis in Section 5.\n- In many practical use cases, probably we already have a decent hand-designed agent at the beginning. If your approach can also effectively improve upon that and is better than other baseline methods, the results will be more solid.\n- The result will look even stronger if you can show your method even beat the previous works on continuous design optimization [2,3] (probably with attribute transform only and no skeleton transform).\n\nTypos:\n- JSMPLs -> JSMLPs in the caption of Figure 4.\n\nReferences:\n- [1] Gupta, Agrim, et al. \"Embodied Intelligence via Learning and Evolution.\" arXiv preprint arXiv:2102.02202 (2021).\n- [2] Luck, Kevin Sebastian, Heni Ben Amor, and Roberto Calandra. \"Data-efficient co-adaptation of morphology and behaviour with deep reinforcement learning.\" Conference on Robot Learning. PMLR, 2020.\n- [3] Schaff, Charles, et al. \"Jointly learning to construct and control agents using deep reinforcement learning.\" 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019.\n\n--------- Post Rebuttal Update ----------\n\nThe authors adequately addressed my main concerns through extra experiments and analysis. \n", "summary_of_the_paper": "This paper proposes a transform-and-control policy to optimize the robotic agents' designs. Contributions include:\n- A novel perspective on agent design: rather than formulating agent design as a bi-level optimization, this paper embeds both design generation and control into a single decision-making process such that both design and control are optimized by the same RL algorithm.\n- In this formulation, the training experience from different designs is shared to improve sample efficiency.\n- Joint-specialized MLP on top of the GNN policy that further finetunes the control of individual joints.", "summary_of_the_review": "This paper provides an interesting perspective on efficient agent design with reasonable technical approaches, and the results look empirically good. I recommend acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "F916S4TWa9t", "reviewer": "Reviewer_487L", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The resulting morphologies of the algorithm seem interesting and effective.\n- The idea of breaking the training into multiple stages where policy first designs the agent and then controls it is novel.\n- Paper is in general well written.\n\nWeaknesses/Questions:\n- Not sure how much control a user has over the design space. For example one may want the design to mimic certain animals, or be symmetrical.\n- The training for the first two stages would involve a delayed sparse reward signal. Is there any intuition why PPO handles this okay in the presented case? Does the horizon of the first two stages, where no reward is given, impact the learning effectiveness?\n- It’d be interested to analyze individual components of the proposed algorithm. E.g. comparing to [1] or [2] assuming a fixed morphology. This will help compare the condition policy approach to a bi-level optimization approach.\n\n[1]. Reinforcement Learning for Improving Agent Design. Ha.\n[2]. Jointly Learning to Construct and Control Agents using Deep Reinforcement Learning. Schaff et al.\n", "summary_of_the_paper": "The paper introduced a reinforcement learning algorithm that simultaneously optimizes the design as well as the controller of a simulated robot to perform locomotion tasks. The core idea is to train a conditioned policy that performs the task in three stages: 1) morphology design of the robot, 2) design parameter adjustment, and 3) controlling of the robot to perform the task. By integrating the design process into the policy learning framework, they are able to design novel and effective agents to complete a variety of tasks. To support the proposed algorithm, graph neural networks is heavily used to support different morphologies. They further propose a joint-specific architecture to improve flexibility of the network, which improves the performance of the algorithm.", "summary_of_the_review": "The paper presents a concrete algorithm with good results in general. It would be helpful to provide some more details and insights in what makes the method work and what are some limitations of the current work.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "05qIgWisM5u", "reviewer": "Reviewer_42PW", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**POSITIVES**\n\nTo the reviewers knowledge (which is admittedly limited), this is the first attempt to learn the morphology of a robot this straight forward with RL. The method is easy to understand and improvements over evolutionary baselines appear very significant (even for 3 seeds).\n\n**NEGATIVES**\n\nThe reviewer has 4 major complaints:\n\n(1) The reward of the tested environments is not scale invariant. When the agent is allowed to construct a robot with twice the dimensions and torque, the robot will move (approximately) twice as fast and therefore get twice the reward. The reviewer assumes that there is a build in penalty for using large torques, but this is a delicate balancing act that the authors need to at least discuss. An additional figure that shows how the robots' size changes during training would reveal a systematic growth of the robot and put this reviewers mind at ease if that is not the case.\n\n(2) The results are based on 3 random seeds. While this is (sadly) all to common in RL, it severely limits the conclusions one can draw with any certainty from the results. The performance gap to the baselines appears large enough to be significant, and the same holds for the \"ours w/o GNNs\" ablation, but the ablations without JSMLP have barely non-overlapping standard deviations, which is an insufficient measure of statistical significance for 3 seeds anyway. To make the presented claims about the ablations, the authors need at least (!) twice the number of seeds (if not more).\n\n(3) The JSMLP method is complicated and poorly explained. It is also not clear to the reviewer why asymmetry is inherently favorable for robotics design. Even if one would like to include the possibility, the presented JSMLP method seems very ad-hoc and might have some unintended side effects. For example, the decision to remove one node can change the node index (e.g. from 31 to 21 in Figure 6) without changing anything about the node. The authors are encouraged to look into Relational GCN as an alternative to represent asymmetric morphologies.\n \n(4) It is unclear whether GNN actually generalize well (in particular out-of-distribution) over different robot morphologies. Kurin et al. (2021) have shown some very convincing counter-examples and suggest to use attention instead of GNN layers. The authors must discuss this alternative and are encouraged to evaluate their experiments with attention layers to see whether this yields any improvement. \n\n**DETAILED COMMENTS**\n\n- $r_t$ must be defined\n- clarify that in the definition of J, H is a variable (due to episodic environment)\n- when you introduce the design D, doesn't it also affect the state and action space?\n- you do not explicitly state in the main paper which variant of GNN you use, e.g. which aggregation function\n- (eq.7) looks as if $a^e$ and $a^d$ can be executed simultaneously instead of alternatively. Better select one $a$ (over the union of the action spaces).\n- it is not clear why you allow the removal of joints, as it could lead to cycles\n- it would be good to have a \"stop changing the morphology\" action instead of always running for $N_s$ and $N_z$ actions\n- the \"Policy Update\" line in Algorithm 1 must be in the first while loop, as PPO first collects the data set M and then uses it to update the model (currently it is called after every episode)\n- you should mention before page 7 that your graph has a root node\n- it would be nice to run in Figure 3 PPO on some standard morphology in each environment, for example HalfCheetah, Ant, Swimmer and Hopper, to see that your algorithm is a \"super-human\" designer. In particular Ant would make for an interesting comparison\n\n**REFERENCES**\n\nKurin, V.; Igl, M.; Rocktaeschel, T.; Boehmer, W., and Whiteson, S. My body is a cage: the role of morphology in graph-based incompatible control. In International Conference on Learning Representations (ICLR), 2021. URL https://openreview.net/forum?id=N3zUDGN5lO.", "summary_of_the_paper": "The paper poses the problem of morphology design for robots as RL training of one joint GNN policy. Their policy first generates the robot's morphology and then evaluates the design with a common behavior policy that conditions on it. The authors also introduce a technique called JSMPL to allow asymmetric morphologies. Experiments in the Mujoco simulator demonstrate a large improvement over evolutionary methods in terms of sample efficiency and final performance (although the latter is less clear, as neither method has clearly converged). Ablation studies show that the GNN architecture is essential, but are less clear about the impact of JSMPL.\n", "summary_of_the_review": "The paper introduces the (to the reviewers knowledge) novel concept of learning the morphology with RL instead of evolution. The experiments need more seeds, but the advantage of the presented method appears significant. The method to allow asymmetric morphologies appears less significant and one might find a more elegant way to ensure this property (if it is at all needed). \n\nThe reviewer would be willing to increase the score if the authors could substantiate point (1) and promise to discuss (or convincingly argue against) the other points of criticism.\n\n**POST-REBUTTAL**\n\nWhile the response does not answer 100% of this reviewers questions, the resulting paper is nonetheless interesting enough, and the results clear enough, to merit publication.  The reviewer is therefore raising the score to 8.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper considers the problem of learning both the physical design (morphology and parameters) of a robot together with the corresponding control policy to optimize performance at a target task. Unlike several contemporary methods that formulate this as two separate, but coupled, optimization problems, the paper unifies these decisions into a single decision-making framework. More specifically, a conditional policy learns to first change an agent's physical design (i.e., the morphology/skeletal structure and its associated parameters), and then to control the design. The policy is formulated as a graph neural network, enabling a single policy to simultaneously control robots with different morphologies (and, in turn, different action spaces). Experimental results demonstrate that the approach outperforms recent baselines on a variety of simulated control tasks.

The paper considers an interesting and challenging problem, that of jointly optimizing an agent's physical design and its control policy, an area of research that has received renewed attention of-late. As the reviewers note, the idea of treating design and control in the context of a single decision-making process is novel. The approach is principled and the experimental results largely justify the significance of the contributions. The reviewers agree that the approach is described clearly and that the paper is well written. The reviewers initially raised a few concerns regarding the experimental evaluation, including the desire for more in-depth evaluations and the need for more random seeds. They also questioned some of the claims made in the initial submission. The authors provided a detailed response to each of these points and made changes to the paper to resolve most of the concerns.

In summary, the paper proposes a novel approach to an interesting problem with convincing results.
Decision: Accept (Oral)