Task-Agnostic Morphology Evolution

1 INTRODUCTION . Recently , deep reinforcement learning has shown impressive success in continuous control problems across a wide range of environments ( Schulman et al. , 2017 ; Barth-Maron et al. , 2018 ; Haarnoja et al. , 2018 ) . The performance of these algorithms is usually measured via the reward achieved by a pre-specified morphology on a pre-specified task . Arguably , such a setting where both the morphology and the task are fixed limits the expressiveness of behavior learning . Biological agents , on the other hand , both adapt their morphology ( through evolution ) and are simultaneously able to solve a multitude of tasks . This is because an agent ’ s performance is intertwined with its morphology as morphology fundamentally endows an agent with the ability to act . But how should one design morphologies that are performative across tasks ? Recent works have approached morphology design using alternating optimization schemes ( Hazard et al. , 2018 ; Wang et al. , 2019 ; Luck et al. , 2020 ) . Here , one step evaluates the performance of morphologies through behavior optimization while the second step improves the morphology design typically through gradient-free optimization . It thus follows that the final morphology ’ s quality will depend directly on the quality of learned behavior as inadequate policy learning will result in a noisy signal to the morphology learner . This begs the question : is behavior learning a necessary crutch upon which morphology optimization should stand ? Unfortunately , behavior learning across a multitude of tasks is both difficult and expensive and hence a precise evaluation of each new candidate morphology requires explicit policy training . As a result , current research on morphology optimization primarily focuses on improving morphology for just one task ( Wang et al. , 2019 ; Ha , 2019 ) . By exploiting task-specific signals , learned morphologies demonstrate impressive performance but provide no guarantees of success outside of the portion of the environment covered by the given task . This is at odds with biological morphologies that are usually able to complete many tasks within their environment . Fundamentally , we want agents that are generalists , not specialists and as such , we seek to shift the paradigm of morphology optimization to multi-task environments . One obvious solution to this issue would be to just learn multiple behaviors in the behavior-learning step . However , such an approach has two challenges . First , multi-task RL is both algorithmically and computationally inhibiting , and hence in itself is an active area of research ( Fu et al. , 2016 ; Yu et al. , 2020 ) . Second , it is unrealistic to assume that we can enumerate all the tasks we would want an agent to perform before its inception . In this work , we propose a framework for morphology design without the requirements of behavior learning or task specification . Instead , inspired by contemporary work in unsupervised skill discovery ( Eysenbach et al. , 2018 ; Sharma et al. , 2019 ) and empowerment ( Mohamed & Rezende , 2015 ) , we derive a task-agnostic objective to evaluate the quality of a morphology . The key idea behind this evaluator is that a performant morphology is likely one that exhibits strong exploration and control by easily reaching a large number of states in a predictable manner . We formalize this intuition with an information-theoretic objective and use it as a fitness function in an evolutionary optimization loop . Candidate morphologies are mutated and then randomly sample and execute action primitives in their environment . The resulting data is used to estimate the agents ’ fitness per the information-theoretic objective . Our contributions are summarized as follows : First , we derive an easily computable informationtheoretic objective to rank morphologies by their ability to explore and control their environment . Second , using this metric in conjunction with Graph Neural Networks , we develop Task-Agnostic Morphology Evolution ( TAME ) , an unsupervised algorithm for discovering morphologies of an arbitrary number of limbs using only randomly sampled action primitives . Third , we empirically demonstrate that across 2D , 3D , and manipulation environments TAME can evolve morphologies that match the multi-task performance of those learned with task supervised algorithms . 2 RELATED WORK . Our approach to morphology optimization builds on a broad set of prior work . For conciseness , we summarize the most relevant ones . Morphology Optimization . Optimizing hardware has been a long studied problem , yet most approaches share two common attributes : first , they all focus on a single task , and second , they all explicitly learn behavior for that task . Sims ( 1994 ) pioneered the field of morphology optimization by simultaneously evolving morphologies of 3D-blocks and their policy networks . Cheney et al . ( 2013 ) and Cheney et al . ( 2018 ) reduce the search space by constraining form and function to oscillating 3D voxels . More recently , Nygaard et al . ( 2020 ) evolve the legs of a real-world robot . Unlike TAME , these approaches depend on task reward as a fitness function to maintain and update a population of agents . Quality diversity based objectives ( Lehman & Stanley , 2011 ; Nordmoen et al. , 2020 ) augment regular task-fitness with unsupervised objectives to discover a diverse population of agents . These approaches are complementary to ours as quality diversity metrics could be incorporated into the TAME algorithm for similar effects . RL has also been applied to optimize the parameters of an agent ’ s pre-defined structure . Ha ( 2019 ) use a population-based policy-gradient method , Schaff et al . ( 2019 ) utilize a distribution over hardware parameters , Luck et al . ( 2020 ) learn a morphology conditioned value function , and Chen et al . ( 2020 ) treat hardware as policy parameters by simulating the agent with computational graphs . While these RL-based approaches explicitly learn task behavior to inform morphology optimization , we do not learn any policies due to their computation expense . Moreover , all these methods are gradient-based , restricting them to fixed topology optimization where morphologies can not have a varying number of joints . Graph Neural Networks . Graph Neural Networks have shown to be effective representations for policy learning across arbitrary agent topologies ( Wang et al. , 2018 ; Huang et al. , 2020 ) . These representations have also been used for agent design . Pathak et al . ( 2019 ) treats agent construction as an RL problem by having modular robots learn to combine . Most related to our work , Neural Graph Evolution ( NGE ) Wang et al . ( 2019 ) evolves agents over arbitrary graph structures by transferring behavior policies from parent to child . Unlike other RL approaches , the use of graph networks allows NGE to mutate arbitrary structures . While these works again are task supervised and only learn morphology for forward locomotion , they inform our use of GNNs to estimate our learning objective . Information Theory and RL . Our information theoretic objective is inspired by several recent works at the intersection of unsupervised RL and information theory . Eysenbach et al . ( 2018 ) and Sharma et al . ( 2019 ) both use information theoretic objectives in order to discover state-covering skills . We apply similar logic to the problem of discovering state-covering morphologies . Gregor et al . ( 2016 ) , Mohamed & Rezende ( 2015 ) and Zhao et al . ( 2020 ) estimate a quantity called “ empowerment ” ( Klyubin et al. , 2005 ) for intrinsic motivation . While empowerment maximizes the mutual information between final state and a sequence of actions by changing actions , we optimize the mutual information over morphologies . Additionally , these metrics for intrinsic motivation require policy learning . More broadly , Oord et al . ( 2018 ) use mutual information to learn representations in an unsupervised manner . 3 METHOD . In this section we introduce our algorithm for Task Agnostic Morphology Evolution ( TAME ) . TAME works by ranking morphologies of arbitrary topology by an information theoretic quantity that serves as proxy for how well an agent can explore and control its environment . Practically , this is accomplished using a GNN to predict the action primitive a morphology executed to reach a specific state . By progressively mutating agents of high fitness according to this metric , TAME discovers morphologies functional over a large portion of the environment without task specification . 3.1 A TASK-AGNOSTIC OBJECTIVE FOR FITNESS . In order to accomplish any task in its environment , a morphology should be able to reliably reach any state through a unique sequence of actions . For example , a morphology that can reach many states but does so stochastically is uncontrollable , while a morphology that can visit a diverse set of states as a consequence of its actions is empowered . We capture this intuition using information theory . Let S , ST , A , and M be random variables representing starting state , terminal state , action primitive , and morphology respectively . Our overall objective is to find morphologies that exhibit high mutual information between terminal states and action primitives , I ( ST ; A|S ) = H ( ST |S ) −H ( ST |A , S ) . Concretely , starting at state S , a good morphology ought to be able to visit a large number of terminal states or equivalently have high entropy H ( ST |S ) . Second , the attained terminal state ST ought to be easily predicted given the action primitive taken A , or H ( ST |A , S ) should be low . As we seek to find morphologies that innately maximize this quantity , our objective becomes argmaxm I ( ST ; A|S , M = m ) . By assuming that all morphologies begin in the same state , we remove the objective ’ s dependence on starting state S and derive a variational lower bound as in ( Barber & Agakov , 2003 ) . argmax m I ( ST ; A|M = m ) = argmax m H ( A|M = m ) −H ( A|ST , M = m ) ≥ argmax m H ( A|M = m ) + Ea∼p ( A|m ) , sT∼p ( ST |a , m ) [ log qφ ( a|sT , m ) ] Note that in the first line we take the dual definition of mutual information , that a morphology M should be able to take as many actions as possible and each of those actions should be predictable from the final state ST . The action distribution depends on the size of the action space of the morphology , thus a ∼ p ( A|m ) and the dynamics also depends on the morphology , thus sT ∼ p ( ST |a , m ) . We then attain a lower bound on our objective by using a classifier qφ ( a|sT , m ) to predict the action primitive taken given a morphology and final state . However , as written this objective still presents a problem : different morphologies have different action spaces , and thus would require different action primitives . We resolve this issue by assuming that each morphology m is composed of k ( m ) joints and that every joint has the same possible set of action primitives . Thus an overall action primitive can be denoted as A = { A1 , ... , Ak ( m ) } and we denote |Aj | as the number of possible primitives for joint j . If we take each joint ’ s action primitive to be independently sampled from a uniform distribution , we can reduce the H ( A|M ) term . We get : argmax m H ( A|M = m ) + Ea∼p ( A|m ) , sT∼p ( ST |a , m ) [ log qφ ( a|sT , m ) ] = argmax m k ( m ) ( log |Aj |+ ( 1/k ( m ) ) Ea∼p ( A|m ) , sT∼p ( ST |a , m ) [ log qφ ( a|sT , m ] ) ≥ argmax m ( k ( m ) ) λ ( log |Aj |+ ( 1/k ( m ) ) Ea∼p ( A|m ) , sT∼p ( ST |a , m ) [ log qφ ( a|sT , m ) ] ) ( 1 ) where 0 ≤ λ ≤ 1 . A full derivation can be found in Appendix A . The resulting objective has two terms . The latter can be interpreted as the average mutual information between the actions of each joint Aj and the final state ST . The former can be interpreted as an adjustment to the overall morphology ’ s information capacity based on the size of its action space or number of joints . Left untouched the objective would grow linearly in number of joints , but logarithmically in the accuracy of qφ . As later detailed in section 3.3 , we only want the best morphologies to have high classification accuracy . As such , we introduce a regularizer λ to attenuate the effect of adding more limbs and emphasize predictability . Similar to empowerment , our objective uses the mutual information between states and actions . However rather than conditioning on starting state and maximizing over a sequence of actions , we maximize with respect to morphology . By assuming the distribution of joint action primitives is uniform , we assume that morphologies ought to use all abilities endowed by their structure . In the next section , we detail how we practically estimate this objective over a set of morphologies .

Summary: This paper develops a general morphology evolution algorithm, and demonstrates its utility in a setting where morphologies are encoded as graphs. The methodology is grounded in a theoretical notion of empowerment, and theory is introduced that extends empowerment to the case of morphology. The morphology evolution itself requires no task-specific signals, but yields morphologies that generalize well to several tasks of interest.