Learning to control self-assembling morphologies: a study of generalization via modularity

1 INTRODUCTION . Only a tiny fraction of the Earth ’ s biomass is composed of higher-level organisms capable of complex sensorimotor actions of the kind popular in contemporary robotics research ( navigation , pick and place , etc ) . A large portion is primitive single-celled organisms , such as bacteria ( Bar-On et al. , 2018 ) . Possibly the single most pivotal event in the history of evolution was the point when single-celled organisms switched from always competing with each other for resources to sometimes cooperating , first by forming colonies , and later by merging into multicellular organisms ( Alberts et al. , 1994 ) . These modular self-assemblies were successful because they combined the high adaptability of single-celled organisms while making it possible for vastly more complex behaviours to emerge . Like many researchers before us ( Murata & Kurokawa , 2007 ; Sims , 1994 ; Tu & Terzopoulos , 1994 ; Yim et al. , 2000 ; 2007 ) , we are inspired by the biology of multicellular evolution as a model for emergent complexity in artificial agents . Unlike most previous work however , we are primarily focused on modularity as a way of improving generalization to novel environmental conditions . In this paper , we present a study of modular self-assemblies of primitive agents — “ limbs ” which can link up to solve a shared task . The limbs have the option to bind together by adding a joint that connects their morphologies ( Figure 1a ) , and when they do so , they pass messages and share rewards . Each limb comes with a simple neural net that controls the torque applied to its joints . Linking and unlinking is treated as a dynamic action , so that the limb assembly can change shape during a single episode of the simulation . This setup has previously been explored in robotics as “ self-reconfiguring modular robots ” ( Stoy et al. , 2010 ) . However , unlike prior work on such robots , where the control policies are hand-defined , we show how to learn the policies and study the generalization properties that emerge . To make this problem computationally tractable , we do not allow the limb assemblies to form cycles in morphology . Limbs pass messages to their neighbors in this graph in order to coordinate behavior . All limbs share a common policy function , parametrized by a neural network , which takes the messages from adjacent limbs as input and outputs a torque to rotate the limb in addition to the linking/un-linking action . We call the aggregate neural network a Dynamic Graph Network ( DGN ) since it is a graph neural network ( Scarselli et al. , 2009 ) that can dynamically change topology as a function of its own outputs . We test our limb assemblies on four tasks : standing , locomotion , pushing and wrestling , shown in Figure 1b . We find that DGNs enable a single modular policy to control multiple possible morphologies , even those unseen during training . For example , a 6-limb policy , trained to build a 6-limb tower , can be applied at test time on 12 limbs , and results in a 12-limb tower . Not only are the policies robust to changes in number of limbs , they also generalize well to novel test-time environmental conditions , such as added wind , or new landscapes . These results together demonstrate that our modular and dynamic self-assembling agents have advantages toward generalization to new environments and tasks . Our main contributions are : • Training primitive agents that self-assemble into complex morphologies to jointly solve control tasks . • Formulating morphological search as a reinforcement learning problem , where linking and unlinking are treated as actions . • Representing policy via a graph whose topology matches the agent ’ s physical structure . • Demonstrating that these self-assembling agents both train and generalize better than fixed- morphology baselines . 2 ENVIRONMENT AND AGENTS . Investigating the co-evolution of control ( i.e. , software ) and morphology ( i.e. , hardware ) is not supported within standard benchmark environments typically used for sensorimotor control , requiring us to create our own . We opted for a minimalist design for our agents , the environment , and the reward structure , which is crucial to ensuring that the emergence of limb assemblies with complex morphologies is not forced , but happens naturally . Environment Structure Our environment contains an arena where a collection of primitive agent limbs can self-assemble to perform control tasks . This arena is a ground surface equipped with gravity and friction . The arena can be procedurally changed to generate a variety of novel terrains by changing the height of each tile on the ground ( see Figure 1b ) . To evaluate the generalization properties of our agents , we generate a series of novels terrains . This include generating bumpy terrain by randomizing the height of nearby tiles , stairs terrain by incrementally increasing height of each row of tiles , hurdles terrain by changing height of each row of tiles , gaps terrain by removing alternate row of tiles , etc . Some variations also include putting the arena ‘ under water ’ which basically amounts to increased drag ( i.e . buoyancy ) . We start our environment with a set of six primitive limb agents on the ground which can assemble to form collectives to perform complex tasks . Agent Structure All our primitive limb agents share the same simple structure : a cylindrical body with a configurable motor on one end . One end of the cylinder is free and the other end contains a configurable motor . The free-end of the limb can link up with the motor-end of the other limb , and then the motor acts as a joint between two limbs with three degrees of rotation . Hence , one can refer to the motor-end of the cylindrical limb as a parent-end and the free end as a child-end . Multiple limbs can attach their child-end to the parent-end of another limb , as shown in Figure 1 ( a ) , to allow for complex graph morphologies to emerge . The limb of the parent-end controls the torques of joint . The un-linking action can be easily implemented by detaching two limbs , but the linking action has to deal with the ambiguity of which limb to connect to ( if at all ) . To resolve these modeling issues , we implement the linking action by attaching the closest limb within a small radius around the parent-node . If no other limb is present within the threshold range , the linking action has no effect . The primitive limb agents are dropped in an environment to jointly solve a given control task . One key component of the self-assembling agent setup that makes it different from typical multi-agent scenarios ( Wooldridge , 2009 ) is that if some agents assemble to form a collective , the resulting morphology becomes a new single agent and all limbs within the morphology maximize a joint reward function . The output action space of each primitive agent contains the continuous torque values that are to be applied to the motor connected to the agent , and are denoted by { τα , τβ , τγ } for three degrees of rotation . In addition to the torque controls , each limb can decide to attach another link at its parent-end , or decide to unlink its child-end if already connected to other limb . The linking and unlinking decisions are binary . This complementary role assignment of child and parent ends , i.e. , parent can only link and child can only unlink , makes it possible to decentralize the control across limbs in a self-assembly . In our self-assembling setup , each agent limb only has access to its local sensory information and does not know about other limbs . The sensory input of each agent includes its own dynamics , i.e. , the location of the limb in 3-D euclidean coordinates , its velocity , angular rotation and angular velocity . Each end of the limb also has a trinary touch sensor to detect whether the end of the cylinder is touching 1 ) the floor , 2 ) another limb , or 3 ) nothing . Additionally , we also provide our limbs with a very simple point depth sensor that captures the surface height on a 9× 9 grid around the projection of center of limb on the surface . One essential requirement to operationalize this setup is an efficient simulator to allow simultaneous simulation of several of these primitive limb agents . We implement our environments in the Unity ML ( Juliani et al. , 2018 ) framework , which is one of the dominant platforms for designing realistic games . For computational reasons , we do not allow the emergence of cycles in the self-assembling agents by not allowing the limbs to link up with already attached limbs within the same morphology . However , our setup is trivially extensible to general graphs . 3 LEARNING TO CONTROL SELF-ASSEMBLING MORPHOLOGIES . Consider a set of primitive limbs indexed by i in { 1 , 2 , . . . , n } , which are dropped in the environment arena E to perform a given continuous control task . If needed , these limbs can assemble to form complex collectives in order to improve their performance on the task . The task is represented by a reward function rt and the goal of the limbs is to maximize the discounted sum of rewards over time t. If some limbs assemble to form a collective , the resulting morphology effectively becomes a single agent with a joint network to maximize the joint reward of the connected limbs . Further , the reward of an assembled morphology is a function of the whole morphology and not the individual agent limbs . For instance , in the task of learning to stand up , the reward is the height of the individual limbs if they are separate , but is the height of the whole morphology if those limbs have assembled into a collective . We now discuss our proposed formulation for learning to control these self-assembling agents . 3.1 CO-EVOLUTION : LINKING/UNLINKING AS AN ACTION . To learn a modular controller policy that could generalize to novel setups , our agents must learn the controller jointly as the morphology evolves over time . The limbs should simultaneously decide which torques to apply to their respective motors , while taking into account the connected morphology . Our hypothesis is that if a controller policy could learn in a modular fashion over iterations of increasingly sophisticated morphologies ( see Figure 3b ) , it could learn to be robust and generalizable to diverse situations . So , how can we optimize control and morphology under a common end-to-end framework ? We propose to treat the decision of linking and unlinking as additional actions of our primitive limb agents . The total action space at at each iteration t can be denoted as { τα , τβ , τγ , σlink , σunlink } where τ∗ denote the raw continuous torque values to be applied at the motor and σ∗ denote the binary actions whether to connect another limb at the parent-end or disconnect the child-end from the other already attached limb . This simple view of morphological evolution allows us to use ideas from learning-driven control , in particular , reinforcement learning ( Sutton & Barto , 1998 ) . 3.2 MODULARITY : SELF-ASSEMBLING AGENT AS A GRAPH OF LIMBS . Integration of control and morphology in a common framework is only the first step . The key question is how to model this controller policy such that it is modular and reuses information across generations of morphologies . Let ait be the action space and s i t be the local sensory input-space of the agent i . One naive approach to maximizing the reward is to simply combine the states of the limbs into the input-space output all the actions jointly using a single network . Formally , the policy is simply ~at = [ a 0 t , a 1 t . . . a n t ] = Π ( s 0 t , s 0 t . . . , s n t ) . This interprets the self-assemblies as a single monolithic agent , ignoring the graphical structure . This is the current approach to solve many control problems , e.g. , Mujoco environments like humanoid ( Brockman et al. , 2016 ) where the policy Π is trained to maximize the sum of discounted rewards using reinforcement learning . In this work , we represent the policy of the agent via a graph neural network ( Scarselli et al. , 2009 ) in such a way that it explicitly corresponds to the morphology of the agent . Let ’ s consider the collection of primitive agent limbs as graph G where each node is denoted by to the primitive limb agent i . Two limbs being physically connected by a joint is analogous to having an edge in the graph . At a joint , the limb which connects itself via its parent-end acts as a parent-node in the corresponding edge , and the other limbs which connect to that joint via child-ends are child-nodes . The parent-node ( i.e. , the agent with the parent-end ) controls the torque of the edge ( i.e. , the joint motor ) , as described in Section-2 .

Summary: The paper considers the problem of constructing compositional robotic morphologies that can solve different continuous control tasks in a (multi-agent) reinforcement learning setting. The authors created an environment where the actor consists of a number of primitive components which interface with each other via "linking" and construct a morphology of a robot. To learn in such an environment, the authors proposed a graph neural network policy architecture and showed that it is better than the baselines on the proposed tasks.