Task-Agnostic Morphology Evolution
Authors: Donald Joseph Hejna III
Pieter Abbeel
Lerrel Pinto
Venue: ICLR 2021 Poster (2021)
Keywords: morphology
unsupervised
evolution
information theory
empowerment

Abstract: Deep reinforcement learning primarily focuses on learning behavior, usually overlooking the fact that an agent's function is largely determined by form. So, how should one go about finding a morphology fit for solving tasks in a given environment? Current approaches that co-adapt morphology and behavior use a specific task's reward as a signal for morphology optimization. However, this often requires expensive policy optimization and results in task-dependent morphologies that are not built to generalize. In this work, we propose a new approach, Task-Agnostic Morphology Evolution (TAME), to alleviate both of these issues. Without any task or reward specification, TAME evolves morphologies by only applying randomly sampled action primitives on a population of agents. This is accomplished using an information-theoretic objective that efficiently ranks agents by their ability to reach diverse states in the environment and the causality of their actions. Finally, we empirically demonstrate that across 2D, 3D, and manipulation environments TAME can evolve morphologies that match the multi-task performance of those learned with task supervised algorithms. Our code and videos can be found at https://sites.google.com/view/task-agnostic-evolution .


Published as a conference paper at ICLR 2021
TASK- MORPHOLOGY EVOLUTION
Donald J. Hejna III UC Berkeley
Pieter Abbeel UC Berkeley
Lerrel Pinto
New York University lerrel0cs. ndu
jhe jeeejnneeeeeee.dddl
palbeee@berreeey.ccm
ABSTRACT
Deep reinforcement learning primarily focuses on learning behavior, usually over- looking the fact that an agent's function is largely determined by form. So, how should one about finding morphology fit for solving tasks in a given environ- ment? Current approaches that co-adapt morphology and behavior use specific task's reward as signal for morphology optimization. However, this often requires expensive policy optimization and results in task-dependent morphologies that are not built to generalize. In this work, we propose new approach, Task- Agnostic Morphology Evolution (TAME), to alleviate both of these issues. Without any task reward specification, TAME evolves morphologies by only applying randomly sampled action primitives on a population of agents. This is accomplished using ceemmaatioo-thooetii objective that efficiently ranks agents by their ability reach diverse states in the environment and the causality of their actions. Finally, we empirically demonstrate that across 2D, 3D, and manipulation environments TAME can evolve morphologies that match the multi-task performance of those learned with task supervised algorithms. Our code and videos can be found https://site //sttes///teeee/ees google.
INTRODUCTION
Recently, deep reinforcement learning has shown impressive success in continuous control problems across wide range of environments (Schulman et al., 2017; Barth-Maron et al., 2018; Haarnoja al., 2018). The performance of these algorithms usually measured via the reward achieved by pre-specified morphology on pre-specified task. Arguably, such a setting where both the morphology and the task are fixed limits the expressiveness of behavior learning. Biological agents, on the other hand, both adapt their morphology (through evolution) and are simultaneously able to solve multitude of tasks. This is because an agent's performance is intertwined with its morphology as morphology fundamentally endows an agent with the ability to act. But how should one design morphologies that are performative across tasks?
Recent works have approached morphology design using alternating optimization schemes (Hazard et al., 2018; Wang et al., 2019; Luck et al., 2020). Here, one step evaluates the performance of morphologies through behavior optimization while the second step improves the morphology design typically through gradient- free optimization. It thus follows that the final morphology's 's quality will depend directly on the quality of learned behavior as inadequate policy learning will result in a noisy signal to the morphology learner. This begs the question: is behavior learning a necessary crutch upon which morphology optimization should stand?
Unfortunately, behavior learning across a multitude of tasks is both difficult and expensive and hence a precise evaluation of each new candidate morphology requires explicit policy training. As result, current research on morphology optimization primarily focuses on improving morphology for just one task (Wang et al., 2019; Ha, 2019). By exploiting task-specific signals, learned morphologies demonstrate impressive performance but provide no guarantees of success outside of the portion of the environment covered by the given task. This is at odds with biological morphologies that are usually able to complete many tasks within their environment. Fundamentally, we want agents that are generalists, not specialists and as such, we seek to shift the paradigm morphology optimization to multi-task environments. One obvious solution to this issue would be to just learn multiple behaviors in the behavior-learnin step. However, such an approach has two challenges. First, multi-task RL is both algorithmically and computationally inhibiting, and hence in itself is active area of research

---

Published as a conference paper at ICLR 2021
VA Diverse, not predictable
Predictubie, but not diverse.
and
Figure/ 1. Visual motivation of our information theoretic objective. We seek evolve morphologies that can easily explore a large number of states and remain predicable while doing so.
(Fu et al., 2016; Yu et al., 2020). Second, it is unrealistic to assume that enumerate all the tasks we would want an agent to perform before its inception.
In this work, propose a framework for morphology design without the requirements of behavior learning or task specification. Instead, inspired by contemporary work in unsupervised skill discovery (Eysenbach al., 2018; Sharma al., 2019) and empowerment (Mohamed & Rezende, 2015), derive a task-agnostic objective to evaluate the quality of a morphology. The key idea behind this evaluator is that a performant morphology is likely one that exhibits strong exploration and control by easily reaching large number of states in a predictable manner. We formalize this intuition with an information-theon theoretic objective and use it as a fitness function in an evolutionary optimization loop. Candidate morphologies are mutated and then randomly sample and execute action primitives in their environment. The resulting data used estimate the agents' fitness per the information-theoretio objective.
Our contributions are summarized as follows: First, we derive an easily computable information- theoretic objective to rank morphologies by their ability to explore and control their environment. Second, using this metric conjunction with Graph Neural Networks, we develop Task-Agnosta Morphology Evolution (TAME), unsupervised algorithm for discovering morphologies of an arbitrary number of limbs using only randomly sampled action primitives. Third, we empirically demonstrate that across 2D, 3D, and manipulation environments TAME can evolve morphologies that match the multi-task performance of those learned with task supervised algorithms.
RELATED WORK
Our approach to morphology optimization builds on broad set of prior work. For conciseness, we summarize the most relevant ones.
Optimizing hardware has been long studied problem, yet most approaches share two common attributes: first, they all focus on single task, and second, they all explicitly learn behavior for that task. Sims (1994) pioneered the field of morphology optimization by simultaneously evolving morphologies of 3D-blocks and their policy networks. Cheney et al. (2013) and Cheney al. (2018) reduce the search space by constraining form and function oscillating 3D voxels. More recently, Nygaard et al. (2020) evolve the legs of a real-world robot. Unlike TAME, these approaches depend task reward fitness function maintain and update population of agents. Quality diversity based objectives (Lehman & Stanley, 2011: 1; Nordmoen et al., 2020) augment regular task-fitness with unsupervised objectives discover diverse population of agents. These approaches are complementary to ours as quality diversity metrics could be incorporated into the TAME algorithm for similar effects. RL has also been applied optimize the parameters of an agent's 's pre-deeiine rre-nefi structure. Ha (2019) use a population- al. (2019) utilize distribution over hardware parameters, Luck al. (2020) learn a morphology conditioned value function, and Chen et al. (2020) treat hardware policy parameters by simulating the agent with computational graphs. While these RL.-based approaches explicitly learn task behavior to inform morphology optimization, we do not learn any policies due to their computation expense. Moreover, all these methods are gradient-basedd restricting them to fixed topology optimization where morphologies cannot have varying number of joints.
Morphology Optimization.
sedicy-gradient method, Schaff
Graph Neural Networks. Graph Neural Networks have shown to be effective representations for policy learning across arbitrary agent topologies (Wang al., 2018: Huang et al., 2020). These representations have also been used for agent design. Pathak et al. (2019) treats agent construction as an RL. problem by having modular robots learn combine. Most related to our work, Neural Graph
2

---

Published as a conference paper at ICLR 2021
Evolution (NGE) Wang al. (2019) evolves agents over arbitrary graph structures by transferring behavior policies from parent to child. Unlike other RL approaches, the use of graph networks allows NGE to mutate arbitrary structures. While these works again are task supervised and only learn morphology for forward locomotion, they inform our use of GNNs to estimate our learning objective.
Information Theory and RL. Our information theoretic objective is inspired by several recent works at the intersection of unsupervised RL and information theory. Eysenbach et al. (2018) and Sharma al. (2019) both use information theoretic objectíves in order to discover state-covering skills. We apply similar logic to the problem of discovering state-covering morphologies. Gregor et al. (2016), Mohamed & Rezende (2015) and Zhao et al. (2020) estimate quantity called "empowernent" (Klyubin et al., 2005) for intrinsic motivation. While empowerment maximizes the mutual information between final state and a sequence of actions by changing actions, we optimize the mutual information over morphologies. Additionally, these metrics for intrinsic motivation require policy learning. More broadly, Oord et al. (2018) use mutual information to learn representations in an unsupervised manner.
METHOD
In this section we introduce our algorithm for Task Agnostic Morphology Evolution (TAME). TAME works by ranking morphologies of arbitrary topology by an information theoretic quantity that serves as proxy for how well an agent can explore and control its environment. Practically, this is accomplished using a GNN to predict the action primitive morphology executed to reach a specific state. By progressively mutating agents of high fitness according to this metric, TAME discovers morphologies functional over a large portion of the environment without task specification.
3.1 A TASK-AGNOSTIC OBJECTIVE FOR FITNESS
In order accomplish any task in its environment, a morphology should be able to reliably reach any state through a unique sequence of actions. For example, a morphology that can reach many states but does so stochastically is uncontrollable, while a morphology that can visit a diverse set of states as a consequence of its actions is empowered. We capture this intuition using information theory. Let S, Sr, A, and M random variables representing starting state, terminal state, action primitive, and morphology respectively. Our overall objective is to find morphologies that exhibit high mutual information between terminal states and action primitives, I T(Sr; AlS) = H(S (S) - H(Sr) A, S). Concretely, starting at state S, good morphology ought to be able to visit large number of terminal states or equivalently have high entropy H (Sr(S). Second, the attained terminal state Sr ought to be easily predicted given the action primitive taken A, or (SS)(A,S) should be low. As we seek to find morphologies that innately maximize this quantity, our objective becomes arg max, (ST; A|S, M m). By assuming that all morphologies begin in the same state, we remove the objective's dependence on starting state S and derive a variational lower bound as in (Barber & Agakov, 2003).
arg max I [SY; AlM m) m
arg max HAAM = m) H(ASST, M m) m
arg max (AAM m) + Euspp A (mm)ssppp((r (a.m) [log (alass, m)]
Note that in the first line we take the dual definition of mutual information, that morphology M should be able to take many actions as possible and each of those actions should be predictable from the final state Sr. The action distribution depends on the size of the action space of the morphology, thus ~ p(Alm) [m) and the dynamics also depends then attain a lower bound on our objective by using a classifier 9, (alsr, Sr. m) to predict the action primitive taken given a morphology and final state. However, written this objective still presents problem: different morphologies have different action spaces, and thus would require different action primitives. We resolve this issue by assuming that each morphology m is composed of k(()) joints and that every joint has the same possible set of action primitives. Thus overall action primitive can be denoted as A = {Ai.... Agtm)) } and we denote |A,1 as the number of possible primitives for joint j. If we take each joint's action primitive to be independently sampled from uniform
the morphology, thus ST ~ p(Srla, m). We
3

---

Published as a conference paper at ICLR 2021
distribution, we reduce the (AAM) term. We get: arg max (AJM = m) + E "awpl//mm)ssspp(Slllm go m)] m
arg max (mm) (log A,1 + (1/k B Ba Am0pppp(mmmm..mmpmpm. [oo (a(a|sT,
m
arg max ((()mm)))((( (log |4,1 + (mm) Baa Banpp(am))ssrsssssssll. [log qo
(1)
where 0 1. A full derivation can be found in Appendix A. The resulting objective has Two terms. The latter can be interpreted the average mutual information between the actions of each joint A, and the final state S7- The former can be interpreted as an adjustment to the overall morphology's information capacity based on the size of its action space or number of joints. Left untouched the objective would grow linearly in number of joints, but logarithmically in the accuracy of 9s. As later detailed in section 3.3, we only want the best morphologies to have high classification accuracy. As such, we introduce a regularizer À to attenuate the effect of adding more limbs and emphasize predictability. Similar to empowerment, our objective uses the mutual information between states and actions. However rather than conditioning starting state and maximizing over sequence of actions, we maximize with respect to morphology. By assuming the distribution of joint action primitives is uniform, assume that morphologies ought to use all abilities endowed by their structure. In the next section, we detail how we practically estimate this objective over a set of morphologies.
3.2 PRACTICALLY ESTIMATING THE OBJECTIVE
In order to estimate the objective in equation 1, we need (a) a representation of morphologies that is able to capture arbitrary topologies, (b) a set of action primitives common across every joint, and (c) a classifier 4s (a) m) that is able to predict said action primitives. We simultaneously address these challenges using graph representations follows:
Morphology Representation. To represent arbitrary morphologies, we follow Sanchez- Sanchez-Gonzaler al. (2018) and Wang al. (2019) and represent each morphology m by a tree T (E, V), where the vertices V represent limbs and the edges E' represent the joints between them. By constructing vertex embeddings i that contain the radius, extent, and position of each limb and edge embeddings es for each limb-connection containing the joint type and range, we define one-to-one mapping between trees 7 and morphologies M.
Action Distribution. Successes in Deep RL (Mnih et al., 2013) have shown that random actions offer sufficient exploration to find solutions to complex tasks. Consequently, we posit that randomly sampled action sequences applied to each of a morphology's joints should be enough to estimate its fitness. However, purely random actions would provide no usable signal to qa- Thus, we define each joint's action space to be a set of predefined action primitives that can each be executed for an extended number of timesteps. In our experiments this takes the form sinusoidal inputs with different frequencies and phases constant torques of differing magnitudes.
GNN Classifier. order to be able to predict the action primitive taken by each joint of varying sized morphologies, we represent q4 as graph neural network classifier trained with cross entropy loss. As we apply actions to joints rather than limbs, we preprocess each morphology tree T by transforming it its line graph G1, where each vertex in Gr is given by edge in E. For edge (i.5) E E, we set the embedding of the line graph vertex to the concatenation (((,)) = [ey, 2y, ((t.)) where 0; Uy are the original node embeddings and e(j)) is the original edge embedding. As we only care about how an agent influences its environment and not its internal state, we also concatenate environment state action taken by the corresponding joint. We implement 9s using Graph Convolutions from Morris G. We then estimate Ellog g (alsT, m)] by summing predicted log probabilities across all joints of a morphology and averaging across all data. We again refer the reader to Appendix A for more mathematical justification.
to each of the node embeddings in Gt. The label for each node in G1 is the
et al. (2019). More details can be found in Appendix
3.3 TASK- MORPHOLOGY EVOLUTION (TAME)
TAME combines our task agnostic objective with a basic evolutionary loop similar to that of Wang et al. (2019) iteratively improve morphology designs from random initialization. While one
4

---

Published as a conference paper at ICLR 2021
might immediately gravitate towards gradient-based approaches, note that arbitrary morphology representations are discrete: the choice to add a limb does not have a computable gradient. We thus turn to evolutionary optimization over population of morphologies P each with fitness f equal to an estimate of the objective in Equation computed using ạs- For each of subsequent generations, do the following:
1. Mutation. If P is empty, we initialize it with randomly sampled morphologies. Otherwise, we sample top morphologies according to the current estimate of the fitness function f. We then mutate these morphologies by randomly adding or removing limbs and perturbing the parameters of some exhisting limbs and joints using Gaussian noise.
2. Data Collection. For each of the newly mutated morphologies, we collect number of short (a1, agim)) ~ p(AAmm) and applying them the
episodes by sampling actions for each joint agent in the environment for a small number of steps. At the end of each episode, we collect the final state 8T ~ p(Sr a, m). We then add the data points (a, sT, m) the overall data set used train lo. and add the new morphologies to the population P.
Update. At the of each generation, we update is by training on all of the collected data. Then, we recompute the fitness of each morphology through equation by evaluating the updated qo on all of the data collected for the given morphology.
Finally, we return the best morphology from the population P according the fitnesses f. Our entire algorithm is summarized in Algorithm and depicted Figure 2. In practice, there are number of additional considerations regarding fitness estimation. In order to get sufficient learning signal, the fitness metric (m) must be varied enough across different morphologies. For example, if ga were able to predict all actions perfectly, all morphologies would have similar fitness. Practically, only the best morphology to be able to achieve high prediction accuracy. To this end we additionally perturb the final states 8r with random Gaussian noise by adding environment randomization where possible. This also forces morphologies to visit more states, as in the presence of noise, actions will need result more distant states be distinguishable.
want
TAME offers a number of benefits. First, we never run any type of policy learning, preventing overfitting to single task and avoiding the slowest step in simulated environments. This makes our approach both more general and faster than task-specifi morphology optimization methods. Second, our method inherently "off "off-policy" with respect morphology, meaning that unlike other evolutionary approaches, we do not need to remove bad solutions from the population P. We maintain all of the data collected and use it to train a better classifier Qá-
Algorithm 1: TAME
(a, Sm.m)
Init. qo (s [8, m) and population P; for i 1,
N do
for j = 1, 2..., L
a (a (alss, m)
mutation from P; for 3: = 1,
do sample joint actions a; 8T + endState(a,, (a, m); PU {m, (m): train(go. P);
Compute fitness
Rollouts with sampled actions
Joint A Action Space
population
for (m. in P do
mutation
f update via equation return arg (m
Figure 2. A visual illustration of the TAME algorithm.
4 EXPERIMENTS
In this section, we discuss empirical results from using TAME to evolve agents in multi-task environ- ments. Note that since there are no standard multi-task morphology optimization benchmarks, we created morphology representation and set of mullii-task environments that will be publicly released. We seek to answer the following questions: Is TAME able to discover good morphologies?; In multi-task settings, how does TAME compare to task supervised algorithms in both performance
5

---

Published as a conference paper at ICLR 2021
and efficiency?; Is TAME's information theoretic objective capable of identifying tegh-perfomiig morphologies? What design choices are critical to TAME's performance?; How important are the action primitives?
4.1 ENVIRONMENT DETAILS
We evaluate TAME on five distinct environments using the MuJoCo simulator (Todorov et al., 2012), each with set of evaluation tasks. We represent morphology tree structure of "capsule" objects with hinge joints.
2D Locomotion: In the 2D locomotion environment, agents are limited to movement in the X-Z plane. The state ST given by the final (r, 2) positions the morphology joints. We evaluate morphologies on three tasks: running forwards, running backwards, and jumping. During evolution we introduce a terrain height-map sampled from a mixture of Gaussians.
Cheetah-Like Locomotion: This is the same the 2D locomotion environment, but starting with an agent similar to the that of the Half Cheetah from DM Control (Tassa et al., 2018). In the cheetah-like environment only, we fix the limb structure of the agent and use multi-layer perceptrons for Is instead of graph convolutions.
3D Locomotion: In the 3D environment, agents can move any direction. The state sT given by the the final (I, y. 2) positions of the joints. We evaluate morphologies on four movement tasks corresponding to each of the following directions: positive X, negative X, positive Y, and negative Y. We again use terrain sampled from mixture of Gaussians.
Arm Reach: Arm agents can grow limbs from fixed point at the center of the X-Y plane. For the reach environment, assess agents by their ability to move the end of their furthest limb to randomly sampled goals within a Im x 1.6m box. The state is the (z, y) position of the end effector.
Arm Push: The push enviornment is the same as the reach environment, except we introduce two movable boxes, and set the state to the (z, y) position of each box. The agent is assessed on six tasks corresponding to moving each of the boxes in one of three directions.
In each environment agents final performance is assessed by the average of its performance each task. In all locomotion experiments, each joint action primitive is given by a cosine function one of possible frequencies (30 ( and one of two possible phases (0 or .T. In all manipulation experiments, each joint action primitive is given by one of four constant torques (-1, -0.5, 0.5, 1). Additional evaluations can be found in Appendix C and additional environment details can be found in Appendix E.
4.2 BASELINES
We compare our method number of different approaches, both task-supervised and task-agnostic.
For this baseline, we randomly sample a morphology train on the tasks. For the "Chectah" finetuning experiment Random corresponds to random mutation of the original design.
Random:
TAMR: Task-Agnostic Morphology Ranking (TAMR) is a variant of our proposed approach without the evolutionary component. Rather than progressively mutating the best morphologies to attain better ones, we generate one large population morphologies, and rank them according to the same fitness function used in TAME.
NGE-Like: We implement a variant of the task supervised Neural Graph Evolution (NGE) algorithm from Wang et al. (2019). In NGE, each mutated morphology inherits a copy of the policy network from its parent and finetunes it, allowing policies to train over time rather than from scratch. This is a much stronger baseline than standard policy learning with MLPs as each new agent finetunes rather than learning from scratch with a very limited number of samples. For a fair comparison we use the same GNN architecture as TAME for the policy networks instead of the NerveNet model (Wang et al., 2018 8) from NGE. In our NGE-like variant we also multi-task
do not use value function based population pruning. We train agent policies environments by giving them a one-hot encoding of the direction for locomotion and the position of the target for manipulation. Fitness is given by the average episode reward over the course of training.

---

Published as a conference paper at ICLR 2021
Env
TAME
TAMR
Random
NGE-Like*
NGE-
Human
Cheetah
1000.0 +
954.7 + 25
974.6 + 42
1012.8
1034.3 52
846.9 + 12
2D
700.4 + 43
467.2 + 36
329.6 +
649.2 + 43
640.8 + 23
846.9 + 12 N/A
3D
533.5 39
348.0
302.3 +
475.7 48
538.3 72
Reach
-0.1
8.1 10
-40.7 26
11
131.5
89.1 1
Push
-295.5 - 16
331.77 1 -449.1 10
144..0
344.31
-388.8 1
Table 1. Full results across all environments and baselines. Top performers excluding the human designed bascline are bolded. * indicates that the given method had access to task rewards during optimization. The human designed baselines were version of the chectah for 2D locomotion and version of the Gym reacher for manipulation.
823.9
613.4
981 33
738.8
764.2
844,5
68.4
54.1
66.6
-205.4
-244.2
-374.7
Figure Selected evolved agents in each of the four main environments. The top row depicts 2D and then 3D Locomotion and the bottom row depicts Arm Reach and then Arm Push from left to right. At the upper right of each agent its average performance across tasks. More agent depictions can be found in Appendix F.
NGK--runnig: This baseline the same as NGE-LLike, except with value based population pruning. This is accomplished by training a separate GNN to predict agent's expected total reward from its morphology embedding. Thompson sampling of the value function (implemented with dropout (Gal & Ghahramani, 2016)) is used to select which mutations to add to the population.
We run all experiments for each environment with the same number of total environment interactions and terrain. For TAME, this amounts to 60 generations of populations of size 24 with 32 short interaction "episodes" for running sampled actions of length 350 for locomotion, 100 for reach, and 125 for pushing. After determining the best morphology, we train it on each of the evaluation tasks with PPO and report the average of the maximum performance across each task. Further experiment details can be found in Appendix G. Each task agnostic method was run for at least twelve seeds. NGE-Like was run for eight seeds.
4.3 RESULTS
Does TAME discover good morphologies? We plot selected results in Figure 4 and list all results in Table 1. Learning curves for TAME can be found in Appendix D. We observe that within error margins, TAME is a top performer in four of the five environments. Qualitatively, TAME produces 2D hopper-like, walker-like, and worm-like agents, interesting yet effective 3D wiggling agents, and functional arms. Depictions of evolved morphologies can be found in Figure 3. While not consistently ahead of human designed robots across all tasks, in all environments some runs matched or surpassed the performance of the human designed robots. Also, TAME able improve upon human designed agents as seen in the Cheetah task.
How does TAME compare to task supervised methods? Across all tasks we find that TAME performs similarly to or surpasses the task supervised NGE-like approach for the same number of timesteps. Only in the cheetah-like environment does the task supervised approach do better, likely because of the strong morphological prior for policy learning. Unlike the policy-learning dependent baselines, TAME able to estimate the quality of a morphology with a very limited number of environment interactions, totaling to around ten fewer task supervised episodes. When additionally considering that each environment is multi-task, the policy learner in the NGE-like approach is left with only handful of rollouts with which optimize each task. This is only exacerbated by the
7

---

Published as a conference paper at ICLR 2021
1030
003
100
-20
830
703
-320
830
-50
-100
430
-420
230
203
-200
-450
-250
7AME
TANS
Bander XCE-UJt
7AME
TAMR
Randoos NGE-LiA
7AME
7AMR
Randon
NGD
TAME
TAMR
Redden 3X0 1Jk
Figure 4. comparison of the performance of TAME versus baselines on four tasks. The vertical line to the right separates the task supervised NGE-like approach. Task-agnostic approaches were run for at least twelve seeds, while NGE-like was run for eight.
challenges of multi-task RL, and the constantly changing dynamics due to morphological mutations. Thus, the NGE-like approach is unlikely to learn an effective policy and provide a strong morphology learning signal. Moreover, NGE only eliminates the bottom portion of the agent population each generation, keeping all but the least performant morphologies around for further policy training. This means that while run of NGE results in both an optimized morphology and policy, for the same number of timesteps it evaluates far fewer designs than TAME, NGE-Pruning only shows modest gains over NGE-like most likely because learning an accurate value function with few, noisy evaluations is difficult.
How fast TAME? Table gives the wall-clock performance different methods in hours. TAME running two CPU cores is consistently faster than the NGE-Like algorithm running eight CPU cores. Normalizing for CPU core count, TAME is 6.78x and 5.09x faster on the 2D and Reach environments respectively. As seek discover morphologies for harder and harder environments, both policy learning and simulation will become increasingly expensive. By requiring no policy optimization and only handful of short environment interactions per agent, TAME will be able to more effectively cover a larger morphological search space in a reasonable amount of time.
2D
3D
Reach
Push
1000
TAME (2)
2.3
3.3
1.1
1.4
TAMR (2)
1.2
2.6
0.4
0.6
NGE-Like (8)
3.9
5.2
1.4
1.8
Rewart
Table 2.
Wallclock performance comparison in hours. Parenthesis indicate CPU cores used.
Alg
2D
3D
Push
200
TAМE
700 + 42
534
+
39
225
16
VarEA
360 + 67
226
35
-293
13
TAME
Global Ac Terrain
Figure 5. Selected ablations on the 2D locomotion environment.
Table 3. Comparison of TAME and a similar al- gorithm with final state variance as the objective.
4.4
UNDERSTANDING TAME'S PERFORMANCE
In this section, we evaluate various ablations TAME to understand which aspects of its design critical its success.
Does TAME's objective correlate with morphology performance? In order assess the quality of our information theoretic objective, we examine two variants of TAME. The first, TAMR, uses the information theoretic objective only to rank population of agents as discussed earlier. TAMR consistently beats random morphologies in terms of final performance, indicating that our objective is indeed selecting good morphologies. Second, TAME with a different metric: the variance final states sT. This has the effect of only measuring fitness by the diversity of states visited and not the controllability of the morphology. Results of
additionally test the performance of variant of
8

---

Published as a conference paper at ICLR 2021
Primitive Type
2D Locomotion
3D Locomotion
Arm Reach
Arm Push
Cosine
700.4 + 42.8
533.5 + 38.7
-0.1 + 14.7
-295.5 + 16.1
Constant
610.4 + 76.6
412.0 36.8
-36.6 + 35.3
-431.7 17.1
Table 4. To test how important selection of the action primitives is in TAME, we compare the final performance of morphologies evolved using different action primitives.
this method on the 2D, 3D, and Push environment can be found in Table 3. In the locomotion environments, this alternative objective fails to create good morphologies.
What design choices in TAME are important? In Figure we provide a number of additional ablations on the specific design choices of TAME that increase the quality of the learned fitness function. First, we try sampling the same action primitive for every joint in morphology. Evidently, by reducing the vocabulary of possible actions, we are unable to get a true sense of how the morphology acts and consequently diminish the accuracy of our fitness metric. This results in much lower performance, approaching that of a random agent. Next, we try removing terrain from the 2D locomotion enviormment, efffcttivlyy reducing the noise in sT. By making actions easier to classify, the spread of fitnesses is reduced, making it harder for evolution to have a strong signal. In TAME, reset 9e every 12 generations to prevent over-fitting older morphologies that have been in the dataset for a large number of epochs. This follows similar logic to replay buffer sampling techniques in off-policy RL (Schaul al., 2015) population pruning in regularized evolution (Real al., 2019). By removing resets, we bias the fitness metric towards older morphologies as do will be better predicting them. We also analyze the effect of the regularizer by setting one. As expected, this degrades performance as high fitness can be attained by simply adding more joints at the expense of predictability, resulting agents that are difficult to control. Finally, the lower performance of TAMR in comparison to TAME across all but the reach task as seen in Table 1 indicate that evolution indeed erucial for finding better solutions.
How important is the choice of action primitives? While TAME does not require task specification, does require specification of a set of per-joint action primitives to use in morphology evaluation. To investigate the importance of choosing apt primitives for given environment, we swap the joint action primitives between the locomotion and manipulation environments, using constant torques for locomotion and sinusoidal signals for manipulation (Table 4). In the locomotion environments, swapping the primitives results in only a modest degredation of performance, and TAME remains competitive with supervised methods. Given that sinusoidal signals are unlikely to appreciably move a robot arm, manipulation agents suffered a larger performance hit. In general, we find that selecting actions primitives can be easily done by visualizing how randomly sampled morphologies behave in the environment under different action sequences. However, there are scenarios where TAME may be less performant due to its use of action primitives. If one component of the state cannot be effected with the chosen vocabulary of action primitives. TAME may not consider it during optimization. TAME also assumes that the starting state the same for every episode and agents. In conjunction with relatively short rollouts, this makes it unlikely that TAME will evolve morphologies that can manipulate components of the state space only present far away from initialization. Despite the use of simple action primitives, morphologies evolved by TAME are general enough to complete multiple tasks in both reaching and locomotion settings. We include additional evaluations on a combined reach and locomotion task in Appendix C.
5 CONCLUSION
We introduce TAME, a method for evolving morphologies without any reward specification policy learning. We show that despite not having access to any explicit behavior signals, TAME is still able to discover morphologies with competitive performance in multi-task settings. As TAME only uses randomly sampled action primitives to estimate an information theoretic objective, it is markedly faster than task supervised approaches. Though simple and fast, the of randomly sampled primitives leaves many open possibilities to be explored in learning morphology through other forms of exploration.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
We thank AWS for computing resources. We also gratefully acknowledge the support from Berkeley DeepDrive, NSF, and the ONR Pecase award. Finally, we would like to thank Stas Tiomkin and the rest of the Robot Learning Lab community for their insightful comments and suggestions.
REFERENCES
David Barber and Felix V Agakov. The im algorithm: variational approach information maximization. In Advances in neural information processing systems, pp. None, 2003. 3
Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva Tb, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. arXiv preprint 1804. 08617, 2018. 1
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv:1606.cm 01540, 2016. 16
Tianjian Chen, Zhanpeng He, and Matei Ciocarlie. Hardware as policy: Mechanical and computa- tional co-optimization using deep reinforcement learning. arXiv preprint wrXiv:2008.0446) 2020.
Nick Cheney, Robert MacCurdy, Jeff Clune, and Hod Lipson. Unshackling evolution: evolving soft robots with multiple materials and powerful generative encoding. In Proceedings of the 15th annual conference on Genetic and evolutionary computation, pp- 167-174, 2013. 2
Nick Cheney, Josh Bongard, Vytas SunSpiral, and Hod Lipson. Scalable co-optimization of morphol- ogy and control in embodied machines. Journal of The Royal Society Interface, 15(143):201709) 2018. 2
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is you need: Learning skills without a reward function. arXiv preprint arXiv: / 1802006070, 2018. 2,3
Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learming on Graphs and Manifolds, 2019.
Justin Fu, Sergey Levine, and Pieter Abbeel. One-shot learning of manipulation skills with online dynamics adaptation and neural network priors. In 2016 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 4019-4026. IEEE, 2016.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059, 2016.7 7
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint arXiv:1611.075000 2016.
David Ha. Reinforcement learning for improving agent design. Artificial life. 2544):352-365 2019.
1.2
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maxi- mum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01299. 2018. 1
Christopher Hazard, Nancy Pollard, and Stelian Coros. Automated design of manipulators for in-hand tasks. In 2018 IEEE-RAS RAS 18th International Conference Humanoid Robots (Humanoids), pp- 1-8. IEEE, 2018.
Wenlong Huang, Igor Mordatch, and Deepak Pathak. One policy to control them all: Shared modular policies for agent-agnostic control. arXiv preprint arXiv:2007.0499 2020. 2
Alexander S Klyubin, Daniel Polani, and Chrystopher L Nehaniv. Empowerment: A universal agent-centric measure control. In 2005 IEEE Congress on Evolutionary Computation, volume 1, pp. 128-135. 135. IEFE, 2005. 3
10

---

Published as a conference paper at ICLR 2021
Joel Lehman and Kenneth O Stanley. Evolving diversity of virtual creatures through novelty search and local competition. In Proceedings of the 13th annual conference on Genetic and evolutionary computation, pp. 211-212111111111 1-218. 2011. 2
Kevin Sebastian Luck, Heni Ben Amor, and Roberto Calandra. Data-efficient co-adaptation of morphology and behaviour with deep reinforcement learning. In Conference on Robot Learning, pp. 854 869. PMLR, 2020. 1,2
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013. 4
Shakir Mohamed and Danilo Jimenez. Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. In Advances neural information processing systems, pp. 2125-2133, 2015. 2,
Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of the AAAI Conference on Artificial Intellignnce, volume 33, pp. 4602-4609, 2019. 4, 20
Jørgen Nordmoen, Frank Veenstra, Kai Olav Ellefsen, and Kyrre Glette. Quality and diversity in
evolutionary modular robotics. arXiv preprint arXiv:2008.02116, 2020. 2
Tonnes F Nygaard, David Howard, and Kyrre Glette. Real world morphological evolution is feasible.
arXiv preprint arXiv:2005.00 09288, 2020. 2
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1 2018. 3
Deepak Pathak, Christopher Lu, Trevor Darrell, Phillip Isola, and Alexei Efros. Learning to control self-assembling morphologies: study of generalization via modularity. In Advances in Neural Information Processing Systems, pp. 2295-2305, 2019. 2
Antonin Raffin, Ashley Hill, Maximilian Ernestus, Adam Gleave Anssi Kanervisto, and Noah htc.hh....ocooooommmmmm /ttp:///RRR//taale--beeeee
Dormann. Stable baselines3. https: petps://githhn /ttp//gttttttt/g// 2019. 20
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. In Proceedings the aaai conference artificial intelligence, volume 33, pp. 4789-4789, 2019. 9
Alvaro Sanzhez-Gonzalez, Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin Riedmiller, Raia Hadsell, and Peter Battaglia. Graph networks as learnable physics engines for inference and control. arXiv preprint arX.v:1806.01242, 2018.
Charles Schaff, David Yunis, Ayan Chakrabarti, and Matthew R Walter. Jointly learning to construct and control agents using deep reinforcement learning. In 2019 International Conference Robotics and Automation (ICRA), pp. 9798-98888 IEEE, 2019.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.088 11100595, 2015. 9
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: arXiv:1707.000 1707.06347, 2017. 1
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. arXiv preprint arXiv: 1907.01177 2019. 2,
Karl Sims. Evolving virtual creatures. In Proceedings of the 2 annual conference on Computer graphics and interactive techniques, pp. 15-22, 1994. 2
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li. Diego de Las Casas, David Budden, Abbas Abdolmalcki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: 1801 00690, 2018. 6, 14
11

---

Published as a conference paper at ICLR 2021
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 0026-5033. IEEE, 2012.
Tingwu Wang, Renjie Liao, Jimmy Ba, and Sanja Fidler. Nervenet: Learning structured policy with graph neural networks. In International Conference on Learning Representations, 2018. 2,
Tingwu Wang, Yuhao Zhou, Sanja Fidler, and Jimmy Ba. Neural graph evolution: Towards efficient automatic robot design. arXiv preprint /rXiv:/906.088
2019. 1, 3, 15
Tianhe Yu, Deirdre Quillen, Zhanpeng Hc, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, pp. 1094-1100, 2020.
Ruihan Zhao, Pieter Abbeel, and Stas Tiomkin. Efficient online estimation of empowerment for reinforcement learning. arXiv preprint arXiv:2007.07356, 2020.
12

---

Published as a conference paper at ICLR 2021
A EXTENDED DERIVATIONS
Let Sr, A, and M be random variables representing terminal state, action primitive, and morphology respectively. Note that A is discrete random variable. We assume morphology is comprised of ((() joints and that the original action primitive space A is symmetric for each joint. Thus, the overall primitive action space A for morphology m can be decomposed into kmm) identical primitive spaces A, for each joint j. Mathematicallyy ( = (A1, Agtm)). ). |Ayl gives the size of the action primitive space for joint j. Assume the distribution of each joint action primitives A, is independent and uniformly distributed. The overall action distribution for morphology by p(A[M m). We now provide the full derivation
objective from Equation
arg max (Sr; A = m)
arg max H (AIM m) H(ASST, M m)
ml arg max HAI m)) + E
pog m)]
2 arg max H(A = m) + E
(al m)]
arg max log A (k(m)
m)]
arg max (m)
log [Ay1 k((m))
Dog M =m)]
1
log [gg1 + 2 arg (m))
golgs M m)]
1((m)
Where S is regularizer on the effect of adding more joints. We can additionally derive per- joint variant of the objective by assuming that the action primitive taken by each joint is conditionally independent given the state and morphology, or p(a|sT ST. m) = p(a, (s,, m) p(a (s, m.. This allows us to use the classification per-joint from do-
arg max(k!())) [og |A,l
pr (log go (alaT, M = -m)]
k((m)
k(m)
arg max(k(mly) log |Ay1+ k(m) Ba Aam.
log II
yo go(asloT, m)
(mm
arg max mmx()
log |Ayl + [((m)
log q ( (ayls2, m)
(mm)
arg max (mm))))
log |Ayl k(m)
Eampp(A m))er=p(Syl..m)
[log go (ST, m)]
The final average and expectation term can be interpreted as the negative cross entropy loss of is joints from morphology m. This is how the value is practically computed in our experiment code.
B RESOURCES
As mentioned in the abstract, our morphology evolution code and videos of learned agents can be found at
C ADDITIONAL EVALUATIONS
We additionally evaluated TAME and our baselines on more complex tasks for the 2D locomotion and 3D locomotion environments to demonstrate learned morphologies adaptable more scenarios.
13

---

Published as a conference paper at ICLR 2021
Method
TAME
TAMR
Random
NGE-Like*
NGE-Pruning
2D Locomotion
113.8 + 15,9
75.3 + 20.1
24.2 6.6
123.8 + 21.9
152.0 21.1
3D Locomotion
7.4 + 1.2
7.7
3.7 1.1
4.2 = 1.4
5.3 + 1.4
Table 5. Additional evaluation of methods goal reaching task in the 2D and 3D environments. * denotes having access to task specification and rewards during morphology optimization.
Rather than just moving in pre-specified direction, we evaluate agents on their ability to reach goals. In 2D the goals are -8m, -4m, 4m, and 8m along the r-axis. In 3D the goals are given by the x-y positions (-4, 0). (0, -4), (4, 0), (0, 4) in meters. The reward is given by the decrease in 72 distance before and after executing given action, and +1 bonus for being within 0. 0. of the goal. We added the goal position minus the current root position to the state space to tell the agent where the goal is. In our evaluation we used the same agents for TAME, TAMR, and Random, but re-ran morphology optimization for the NGE-like and
methods as the reward function changed. Results given Table 4, showing the average reward across policies trained independently for each goal via million steps of PPO, unlike in other locomotion experiments which were run for 500,000 steps. We again find that TAME is within confidence bounds of the regular NGE-like method despite not having access to task specification rewards, though is slightly outperformed by the NGE-Pruning method. In 3D, TAME does better than the task-supervised methods, likely because policy learning much more difficult 3D resulting in worse morphology signal for the NGE style baselines. Additionally, we investigate the importance of reward and state space specification for the NGE-lik, lik-like methods. Rather than providing the difference between the goal position and the current position to the agent, we re-ran the 21) experiments but instead provided one-hot encoding of the goal and the absolute position of the agent. In this setting, performance on the NGE baselines dropped to 72.8 + 24.6 and 95.0 + 20.5 for NGE-like and NGE-Pruning respectively. This is likely because it is harder to re-use information across tasks when the goal is provided as a one-hot vector instead of relative position.
D LEARNING CURVES
Here we provide the learning curves for TAME.
Fitness Curves for Locomotion Environments
Fitpess Curves for Manipulation Eavironments
144
1.2
0.8
Fitness
0.8
0.4
2D Locomotion
Locomation
Block Environment
Cheetake like Locomotion
Reach Environment
generation
generation
Figure Learning curves plotting the TAME information theoretic objective during evolution. Error bars the standard deviation across seeds. Note that qs reset every 12 generations.
E ENVIRONMENT DETAILS
Here we provide additional specification on the environments used to assess TAME. Images of the environments used during evolution be found in Figure 7.
Morphology Representation. We implement morphologies using tree like objects and MJCF in Deepmind's DM Control Suite (Tassa et al., 2018). Morphologies are composed of nodes which all
14

---

Published as a conference paper at ICLR 2021
(a) A random morphology in the 2D (b) A random morphology in the 3D (c) A random morphology in the block environment.
environment
environment
Figure 7. Images of all environments used in experiments.
take the form of "capsule" objects. To fully specify a node, one must provide its radius, x-length, y-length, and z-length. In 2D and Arm environments, we restrict lengths in some directions to be zero. Nodes are connected to one another using fixed or hinge joints. Given a morphology represented by tree, connection between nodes will always have parent and child. A joint fully specified by a type (fixed or hinge on relative z or y axis of the parent), an attachment point from zero to one that specifies how far down the extent of the parent node the child node attaches, symmetric joint range, and a joint strength or gear.
An additional consideration with this arbitrary morphology representations collisions. Without collision detection, limbs would be allowed to grow into each other, resulting in unrealistic morpholo- gies and unstable simulations. While previous works that MuloCo for morphology optimization circumvent this problem by disabling contacts between the limbs of morphologies (Wang et al., 2019), we develop a contact detection system that prevents random sampling or mutations from creating invalid structures. As such, all of our environments have collisions between limbs enabled.
Morphology Sampling. Random morphologies are recursively sampled BFS order. First, we randomly generate root node by sampling parameters uniformly over allowed ranges and add it to node queue. While the node queue non-empty and we are under the maximum allowed number of limbs, we pop a parent node from the queue, run a Bernoulli trial to determine if it will grow child node, and enqueue sampling of designs, for sampling and then repeatedly mutate the agent until it has that number nodes.
randomly sampled child node created. In order to allow for more uniform agents first sample a number of limbs uniformly random,
Morphology Mutation. We mutate morphologies in three steps. First, we iterate over all of the nodes of a morphology and with some probability perturb its parameters with Gaussian noise. Note that we consider the edge or joint attributes be part of the child, and perturb these parameters with those of the child's limb. Next, we collect all the nodes that have not maxed out their child count, small probability. Finally, we collect all the leaf
and have each of them grow child node with nodes of the tree, and delete each of them with some small probability.
Locomotion Details. In locomotion environments agents are tasked with moving different direc- tions. We use physics parameters similar to those of the DM Control cheetah, though we remove the total-mass specification. We restrict capsules be most length 0.75m in each direction and have radius between 0.035m and 0.07m. Joints are restricted to having a range of 30 and 70 degrees in either direction with a gear between 50 and 100. In the 2D environments, we restrict the agent to only have limbs grow in the X and Z. directions and have joints on the Y-axis. Fach node in the morphology tree allowed have at most two children, and during random generation children are sampled with probability 0.3. Parameters of each node are perturbed with probability of 0.14 for the geometric attributes and 0.07 for the joint attributes. During evolution, we use a standard deviation of 0.125 in modifying the lengths of the nodes, add limb with probability 0.I, and remove limb with probability 0.08. We also force all edges to be joints and do not allow static connections, except in the cheetah environment.
The reward for the different movement tasks in the X, negative X, Y, and negative directions is given by min(10 10, max(-10) 10, v)), where is the agent's linear velocity in the desired direction. For
15

---

Published as a conference paper at ICLR 2021
the Z movement task, the reward is given by min max(-2, v)). In the main body of the paper, we refer to X, negative X, and Z as forward, backward, and jumping respectively.
Finally, sample terrain height maps using random Gaussian mixtures as depicted in Figure 8. For the 2D environments, the maximum terrain height is 0.2m, while for the 3D environments, is 0.25. However, the variance for the sampled 2D Gaussians is restricted to be between 0.2m and 0.8m, making the 2D terrain bumpier than the 3D terrain that results from randomly sampled covariance matrices. For locomotion tasks we additionally introduce Gaussian state noise of mean zero and standard deviation 0.05.
Figure 8, Images of random agents in 2D and 3D terrain respectively.
Manipulation Details In manipulation environments, arm-like agents either reach goals attempt to push blocks to specified locations, For these environments, we use similar parameters to that of the Open AI Gym reacher (Brockman et al., 2016), though with scaled up bodies. During the construction phase, we limit the limbs of arm-agents to only grow the direction to a maximum length of 0.75 and radius between 0.04m and 0.07m. The base of the arm is fixed (0,0). Joints between limbs of the arm are restricted to the Z-axis and have ranges hetween 90 and 175 degrees in either direction and gearings between 70 and 80. Nodes in the environment can only have one child. They grow or remove children with probability 0.2. The perturbation probabilities are the same as locomotion.
The reward for the reach task is given by the negative 12 distance between the end of the agent's further limb and the pre-selected goal position from box of size Im by 1.6m. A bonus of 100 is given when the agent is within 0.1m of the desired position. The push task is more complicated. Two 0.2m boxes, box and box are introduced at (0.9. 0.65) and (0.9, 0-0.65) respectively. There are six total pushing tasks that result from moving each of the boxes to one of three positions. Box is pushed to (0.0, 0.65), (0.9, 1.4), and (0.15, 1.25). Box pushed to (0.0, -0.65), (0.9, -1.4), and (0.15, 1.25). The reward for the box pushing task is given by - -lobox target||2 0.5| box arm] 2
For manipulation tasks we additionally introduce Gaussian state noise of mean zero and standard deviation 0.2.
16

---

Published as a conference paper at ICLR 2021
F ADDITIONAL MORPHOLOGY RESULTS
Here provide images of additional morphologies constructed using algorithms and baselines.
Σ
1
4
K
X
K
7
r
^
Figure 9, Morphologies evolved by TAME on all tasks.
17

---

Published as a conference paper at ICLR 2021
4
4
7
A
1
1
4
4
A
X
Figure 10. Morphologies evolved by TAMR on all tasks.
18

---

Published as a conference paper at ICLR 2021
Figure 1I. Randomly sampled morphologies for each task.
19

---

Published as a conference paper at ICLR 2021
4
A
R
V
Figure 12. Morphologies evolved by the NGE-Like algorithm all tasks.
G ARCHITECTURE AND HYPERPARAMETERS
We implement all graph models using Graph Convolutions from the Pytorch Geometric software package (Fey & Lenssen, 2019). Graph convolutions are able to develop global features by passing messages between nodes, Each node in the graph computes a message via fully connected layer and a state using different fully connected layer. Next, each node averages all of its incoming messages and adds it its state. For more detail, refer the reader to Morris et al. (2019). All of our graph networks for both TAME and NGE-like have three Graph Convolutions and use ReLU activations. For policy evaluations,
the PPO implementation from Stable- Barlle-BBsiies 3 (Raffin et al., 2019).
Below we provide list all hyper-paramettee parameters. Note that all experiments were run for the same total number of timesteps mentioned in Section 4. Thus, for the NGE-like baseline, each individual ran policy learning for 32 x 350 = 1120 timesteps per generation in the locomotion tasks. The same relationship holds for manipulation. Table 6 lists hyperparameters used for evaluations. Table lists hyperparameters used for evolution.
Hyperparameter
Value
Discount
0.99
GAE Parameter
0.95
steps per udpate
1000
epochs per update
batch size learning rate # workers entropy coef Hidden Layers
128
0.0003
0.001
2xDense256
Table 6. PPO Hyperparameters used for all policy evaluations. For each locomotion task, trained policies for 500k timesteps per task. For the arm reach task, we trained policies for one million timesteps, and for the push task trained policies for 200k timesteps per task.
20

---

Published as a conference paper at ICLR 2021
Hyperparameter
Value
generations
60
population size
24
episodes/individuall
32
learning rate
0.001
batch size epochs/gen 0.25, 0.2
128
15
lambda
hidden layers
3xGraphConv192
reset freq prediction classes
12
4
mutate
0.06
Table 7. Evolution hyperparameters. If two values are listed, the first set are for locomotion environments, the second are for manipulation. In practice we additionally take the natural log of k(m)), Mutate % refers to the percentage of the population we randomly sample morphologies from. For NGE-Like, maintain 60% of the population each generation, and sample new 40% from the existing population.
21


Reviews:
[{"review_id": "jCBAeT1NWC8", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a method to evolve morphologies of a robotic system without explicit reward and using an empowerment-like quantity. The thus obtained morphologies can get higher rewards in task settings when RL algorithms are applied.\n\nThe story and presentation of the paper are clear. I also like the results and think it is interesting, maybe more for a conference like ALife than ICLR though.\n\nStrengths:\n- Interesting way to estimate the information criterion using a GNN\n- formulation of a morphology-empowerment\n- analysis and ablations justify the design choices and the method\n- good results (on self-given tasks)\n\nWeaknesses:\n- the mathematical formulation is sloppy in many places\n- the morphologies are compared on different sequences. It might be interesting to know how much variance the estimation of Eq 1 has when sampling a different batch of action-sequences\n\nI think when the problems with the formulation are fixed and the typos are removed the paper can be much stronger.\n\nRelated work: As you quantity is very close to the Empowerment definition, I think the original work by the Polani group should be cited, e.g. \"Empowerment: A universal agent-centric measure of control\", Klyubin, Polani, Nehaniv\nAn interesting combination could be to use task agnostic live-time adaptation, such as predictive information maximization (Information Driven Self-Organization of Complex Robotic Behaviors, PLoSOne) or related work could be combined.\n\nProblems in the mathematical formulation:\nPage 3, first equation (before (1)): What is the expectation taken over? \nYou write about q_theta being a classifier, but most of the paper reads like you have continuous actions. This should be clarified early on that you assume discrete actions/ action-primitives for finding the morphologies.\nEq 1: j appears in the right but without any specification. \nYou write that you take expectations of joints, but what is p(j), so the probability of a joint. I think you want to add a sum over the joints $j$ or something. What is |J|?\n\n\nDetails:\n- Sec 1: Second, Using (capital)\n- Sec 3.1: Notice that left untouched, the...\n- Sec 3.1: By assuming actions are uniform A_j?\n- Action Distr: The appendix should contain information about the action sequences you are using.\n- GNN Classifier: the the \n- Sec 3.3: slowest step in for simulation....\n- Sec 4.4: \"meta\" action: use the same label as in Fig 5  (global action)\n- Table 3: ..TAME and a similar \n- Sec 5: You write randomly sampled actions, but you have highly structured action-primitives \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "iFqbwfl-Hp", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper develops a general morphology evolution algorithm, and demonstrates its utility in a setting where morphologies are encoded as graphs. The methodology is grounded in a theoretical notion of empowerment, and theory is introduced that extends empowerment to the case of morphology. The morphology evolution itself requires no task-specific signals, but yields morphologies that generalize well to several tasks of interest.\n\n\nStrong points:\n\nThe extension of empowerment to morphologies is novel, interesting, and well-motivated.\n\nThe use of random actions is satisfying in that it is initially counter-intuitive, but well-grounded, and in line with other recent advances based in generating interesting things from random input (e.g., GANs).\n\nThe introduced space of morphologies is quite general and can capture some very interesting designs.\n\nThe ablations in Figure 5 and Table 3 are very helpful in validating the theory. Table 3 in particular verifies how empowerment can provide an advantage over more naiive diversity metrics.\n\n\n\n\nWeak points:\n\nThere are some key gaps in related work literature.\n\nOne is NS-LC (Evolving a Diversity of Creatures through Novelty Search and Local Competition,), which was used to evolve explicitly diverse high-performing morphologies (and is an example of the broader category of quality diversity algorithms). The goal of that work is different: to collect diverse morphologies, instead of finding one that is maximally general, but the methodology is related. There is generally a lack of references in the description of the evolutionary algorithm (EA) used in the paper. Is this similar to algorithms used to evolve other sorts of open-ended structures? E.g., open-ended neural network design (e.g., Evolving Deep Neural Networks)? Or EAs that include generation-by-generation updates of a model like the classifier used in this paper? E.g., surrogate-assisted methods? \n\nThere is also a missing link to recent advances in self-supervised methods.\n\n“we reset q_phi every 12 generations to prevent over-fitting to older morphologies” Worth mentioning the connection of this to ideas like experience replay in RL and removing the oldest individual in Regularized Evolution.\n\nThe theory motivates the use of random actions, but the experiments use a highly restricted set of actions that are well-suited to the eventual tasks of interest, which takes away from the promise of generality and task-agnosticism of the method. Should we expect these specialized “predefined action sequences” to generalize to a wide variety of morphologies and tasks, or does the experimenter need to design these by hand depending on the tasks?\n\n“each joint action is given by a cosine function of one of two possible frequencies and one of two possible phases”. What are these frequencies and phases? So, each joint has 4 possible controllers. Are combinations of joints with these controllers sufficient for performing arbitrary tasks?\n\nSimilarly, the tasks in the paper are also closely-aligned with the empowerment objective. That is, the state space is low-dimensional, and task is measured in a 1- or 2-dimensional slice of the state space. Can we expect the approach could generalize to more complex tasks? E.g., can the approach handle a task that combines locomotion and reach, i.e., navigate to randomly sampled goals when the agent is not tethered? Why not evolve a single morphology that can solve all the classes of tasks in experiments (except cheetah)? Isn’t this the promise of the method?\n\nThe changes from NGE to NGE-like make the method more comparable to TAME, but the paper should elaborate on what the missing parts do and whether TAME could take advantage of them. E.g., what is “value based population pruning” and why doesn’t TAME use it? What is NerveNet and why doesn’t TAME use it? Why was NGE-like run for 8 seeds instead of 12? Because it’s slower? Why are NGE-like and TAME run on different numbers of cores?\n\nOne key baseline missing is single-task or multi-task version of the EA that TAME uses, but simply using task fitness instead of empowerment. This would give a sense of how the task-specific variant of your underlying EA compares to NGE, and validate whether the EA is at least powerful enough to take advantage of task signals when they are available.\n\n\n\nMinor comments:\n\n“Additionally, morphologies with better exploration will be able to learn to solve tasks more quickly.” Is there existing work that shows this or some other explanation? It makes intuitive sense, but may not be true in general.\n\nIn Eq. 1, looks like “log|J|” should be “log|A_j|”, and the expectation “E” should be formatted as in the previous lines.\n\nShould TAME Cheetah be bold in Table 1 since it is within the margin of NGE-like?\n\nIn “How does Tame compare to task supervised methods?”: “We hypothesize that this” -> Clarify what “this” is and why NGE-like is good on Cheetah (because the morphology space is more limited or structured?).\n\n-------\nBased on the author response, the rating has been updated (see comments below).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "8kcoZqBqvWU", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces an algorithm for optimizing the robot morphology in a simulated environment. The key idea is that instead of finding a morphology and a controller for a specific task, they propose to search for a morphology that can reach a large variety of states in a predictable way. Specifically, they developed an objective function that maximizes the mutual information between the actions and the final states of the robot. The proposed algorithm is demonstrated on a few simulated locomotion and manipulation tasks.\n\nThe paper is well written and the problem that the work is addressing is significant in my point of view.\n\nWhat I like about the work:\n1. The idea of optimizing morphology in a task-agnostic way is interesting.\n2. The proposed formulation and algorithm seems concrete and reasonable.\n3. The proposed algorithm seems effective for the designed experiments presented and compares favorably to baseline methods.\n\nMy concerns about the work:\n1. The proposed objective only considers reachability of the robot, but not the efficiency of completing the task. A good baseline would be whether the optimized design works better than some human-designed morphology. For example, does the optimized result outperform the hopper, halfcheetah, and the 2d manipulator from OpenAI gym environments?\n2. Is the algorithm sensitive to the choice of the regularizer lambda that balances adding limbs and prediction accuracy?\n3. The actions used to explore the environment is quite simplistic (sinusoidal-based for locomotion and constant for manipulation). Though it works for the presented problems, it's not clear if it will work for more complex problems. For example, if we want to design a biped robot that does not fall to the ground, random exploration would likely give us a very narrow range of states.\n\nIn general I think the paper introduces an interesting idea. It would be a good contribution to ICLR if my concerns above could be addressed.", "extra_scores": {}, "extra_text": {}}, {"review_id": "0LLyovvqIv", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an approach that evolves morphologies for virtual creatures that can reach as many states in their environment as possible. The approach is based on an information-theoretic objective that rewards morphologies for reaching diverse states and at the same time performing behaviours that are predictable. \n\nWhile the paper is interesting there are a few issues that should be addressed:\n- The approach relies on randomly sampled actions. How would it scale to more complex tasks that require non-random actions to be solved? Additionally, most of the morphologies are relatively simple, compared to some of the other work in evolving virtual creatures. Is it possible to scale the approach to produce more complex morphologies?\n- \"This is at odds with biological morphologies that seek to be functional over the largest subset of their environment as possible. \" -> I would argue the opposite. Biological morphologies are very much specialist to work well in their particular niche. \n- The approach does not seem completely task-agnostic since the actions are sampled from different distributions for each task (e.g. sine/cosine). \n- The disadvantages and limitations of the approach should be discussed in more detail\n- \"We implement a variant of the task supervised Neural Graph Evolution (NGE) algorithm from Wang et al. (2019) without value based population pruning to better isolate the effects of reward supervision. “ -> how well does it work with value-based population training?\n- How does the approach compare to quality diversity approaches that also aim to produce many morphologies in a single run, e.g. Novelty Search with Local Competition (\"Evolving a Diversity of Virtual Creatures through Novelty Search and Local Competition\" Lehman et al.)? \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors use Empowerment for morphology optimisation, a quite novel idea. After initial unclarities and various improvements on the submission, the reviewers unanimously voted for acceptance of the paper.  
Decision: Accept (Poster)