Neural Graph Evolution: Towards Efficient Automatic Robot Design

1 INTRODUCTION . The goal of robot design is to find an optimal body structure and its means of locomotion to best achieve a given objective in an environment . Robot design often relies on careful human-engineering and expert knowledge . The field of automatic robot design aims to search for these structures automatically . This has been a long-studied subject , however , with limited success . There are two major challenges : 1 ) the search space of all possible designs is large and combinatorial , and 2 ) the evaluation of each design requires learning or testing a separate optimal controller that is often expensive to obtain . In ( Sims , 1994 ) , the authors evolved creatures with 3D-blocks . Recently , soft robots have been studied in ( Joachimczak et al. , 2014 ) , which were evolved by adding small cells connected to the old ones . In ( Cheney et al. , 2014 ) , the 3D voxels were treated as the minimum element of the robot . Most evolutionary robots ( Duff et al. , 2001 ; Neri , 2010 ) require heavy engineering of the initial structures , evolving rules and careful human-guidance . Due to the combinatorial nature of the problem , evolutionary , genetic or random structure search have been the de facto algorithms of automatic robot design in the pioneering works ( Sims , 1994 ; Steels , 1993 ; Mitchell & Forrest , 1994 ; Langton , 1997 ; Lee , 1998 ; Taylor , 2017 ; Calandra et al. , 2016 ) . In terms of the underlying algorithm , ∗Two authors contribute equally . most of these works have a similar population-based optimization loop to the one used in ( Sims , 1994 ) . None of these algorithms are able to evolve kinematically reasonable structures , as a result of large search space and the inefficient evaluation of candidates . Similar in vein to automatic robot design , automatic neural architecture search also faces a large combinatorial search space and difficulty in evaluation . There have been several approaches to tackle these problems . Bayesian optimization approaches ( Snoek et al. , 2012 ) primarily focus on fine-tuning the number of hidden units and layers from a predefined set . Reinforcement learning ( Zoph & Le , 2016 ) and genetic algorithms ( Liu et al. , 2017 ) are studied to evolve recurrent neural networks ( RNNs ) and convolutional neural networks ( CNNs ) from scratch in order to maximize the validation accuracy . These approaches are computationally expensive because a large number of candidate networks have to be trained from grounds up . ( Pham et al. , 2018 ) and ( Stanley & Miikkulainen , 2002 ) propose weight sharing among all possible candidates in the search space to effectively amortize the inner loop training time and thus speed up the architecture search . A typical neural architecture search on ImageNet ( Krizhevsky et al. , 2012 ) takes 1.5 days using 200 GPUs ( Liu et al. , 2017 ) . In this paper , we propose an efficient search method for automatic robot design , Neural Graph Evolution ( NGE ) , that co-evolves both , the robot design and the control policy . Unlike the recent reinforcement learning work , where the control policies are learnt on specific robots carefully designed by human experts ( Mnih et al. , 2013 ; Bansal et al. , 2017 ; Heess et al. , 2017 ) , NGE aims to adapt the robot design along with policy learning to maximize the agent ’ s performance . NGE formulates automatic robot design as a graph search problem . It uses a graph as the main backbone of rich design representation and graph neural networks ( GNN ) as the controller . This is key in order to achieve efficiency of candidate structure evaluation during evolutionary graph search . Similar to previous algorithms like ( Sims , 1994 ) , NGE iteratively evolves new graphs and removes graphs based on the performance guided by the learnt GNN controller . The specific contributions of this paper are as follows : • We formulate the automatic robot design as a graph search problem . • We utilize graph neural networks ( GNNs ) to share the weights between the controllers , which greatly reduces the computation time needed to evaluate each new robot design . • To balance exploration and exploitation during the search , we developed a mutation scheme that incorporates model uncertainty of the graphs . We show that NGE automatically discovers robot designs that are comparable to the ones designed by human experts in MuJoCo ( Todorov et al. , 2012 ) , while random graph search or naive evolutionary structure search ( Sims , 1994 ) fail to discover meaningful results on these tasks . 2 BACKGROUND . 2.1 REINFORCEMENT LEARNING . In reinforcement learning ( RL ) , the problem is usually formulated as a Markov Decision Process ( MDP ) . The infinite-horizon discounted MDP consists of a tuple of ( S , A , γ , P , R ) , respectively the state space , action space , discount factor , transition function , and reward function . The objective of the agent is to maximize the total expected reward J ( θ ) = Eπ [ ∑∞ t=0 γ tr ( st , at ) ] , where the state transition follows the distribution P ( st+1|st , at ) . Here , st and at denotes the state and action at time step t , and r ( st , at ) is the reward function . In this paper , to evaluate each robot structure , we use PPO to train RL agents ( Schulman et al. , 2017 ; Heess et al. , 2017 ) . PPO uses a neural network parameterized as πθ ( at|st ) to represent the policy , and adds a penalty for the KL-divergence between the new and old policy to prevent over-optimistic updates . PPO optimizes the following surrogate objective function instead : JPPO ( θ ) = Eπθ [ ∞∑ t=0 At ( st , at ) r t ( st , at ) ] − β KL [ πθ ( : |st ) |πθold ( : |st ) ] . ( 1 ) We denote the estimate of the expected total reward given the current state-action pair , the value and the advantage functions , as Qt ( st , at ) , V ( st ) and At ( st , at ) respectively . PPO solves the problem by iteratively generating samples and optimizing JPPO ( Schulman et al. , 2017 ) . 2.2 GRAPH NEURAL NETWORK . Graph Neural Networks ( GNNs ) are suitable for processing data in the form of graph ( Bruna et al. , 2014 ; Defferrard et al. , 2016 ; Li et al. , 2015 ; Kipf & Welling , 2017 ; Duvenaud et al. , 2015 ; Henaff et al. , 2015 ) . Recently , the use of GNNs in locomotion control has greatly increased the transferability of controllers ( Wang et al. , 2018 ) . A GNN operates on a graph whose nodes and edges are denoted respectively as u ∈ V and e ∈ E. We consider the following GNN , where at timestep t each node in GNN receives an input feature and is supposed to produce an output at a node level . Input Model : The input feature for node u is denoted as xtu . xtu is a vector of size d , where d is the size of features . In most cases , xtu is produced by the output of an embedding function used to encode information about u into d-dimensional space . Propagation Model : Within each timestep t , the GNN performs T internal propagations , so that each node has global ( neighbourhood ) information . In each propagation , every node communicates with its neighbours , and updates its hidden state by absorbing the input feature and message . We denote the hidden state at the internal propagation step τ ( τ ≤ T ) as ht , τu . Note that ht,0u is usually initialized as ht−1 , Tu , i.e. , the final hidden state in the previous time step . h 0,0 is usually initialized to zeros . The message that u sends to its neighbors is computed as mt , τu = M ( h t , τ−1 u ) , ( 2 ) where M is the message function . To compute the updated ht , τu , we use the following equations : rt , τu = R ( { mt , τv | ∀v ∈ NG ( u ) } ) , ht , τu = U ( ht , τ−1u , ( rt , τu ; xtu ) ) ( 3 ) whereR and U are the message aggregation function and the update function respectively , andNG ( u ) denotes the neighbors of u . Output Model : Output function F takes input the node ’ s hidden states after the last internal propagation . The node-level output for node u is therefore defined as µtu = F ( h t , T u ) . Functions M , R , U , F in GNNs can be trainable neural networks or linear functions . For details of GNN controllers , we refer readers to ( Wang et al. , 2018 ) . 3 NEURAL GRAPH EVOLUTION . In robotics design , every component , including the robot arms , finger and foot , can be regarded as a node . The connections between the components can be represented as edges . In locomotion control , the robotic simulators like MuJoCo ( Todorov et al. , 2012 ) use an XML file to record the graph of the robot . As we can see , robot design is naturally represented by a graph . To better illustrate Neural Graph Evolution ( NGE ) , we first introduce the terminology and summarize the algorithm . Graph and Species . We use an undirected graph G = ( V , E , A ) to represent each robotic design . V and E are the collection of physical body nodes and edges in the graph , respectively . The mapping Algorithm 1 Neural Graph Evolution 1 : Initialize generation P0 ← { ( θ0i , G0i ) } Ni=1 2 : while Evolving jth generation do . Evolution outer loop 3 : for ith species ( θji , G j i ) ∈ P j do . Species fitness inner loop 4 : θj+1i ← Update ( θ j i ) . Train policy network 5 : ξi ← ξ ( θj+1i , G j i ) . Evaluate fitness 6 : end for 7 : Pj+1 ← Pj \ { ( θjk , G j k ) ∈ P j , ∀k ∈ arg minK ( { ξi } ) } . . Remove worst K species 8 : P̂ ← { ( θ̂h , Ĝh =M ( Gh , p ) ) , whereGh , p ∼ Uniform ( Pj+1 ) } Ch=1 . Mutate from survivors 9 : Pj+1 ← Pj+1 ∪ { ( θ̂k , Ĝk ) ∈ P̂ , ∀k ∈ arg maxK ( { ξP ( Ĝh ) } ) } . . Pruning 10 : end while A : V → Λ maps the node u ∈ V to its structural attributes A ( u ) ∈ Λ , where Λ is the attributes space . For example , the fish in Figure 1 consists of a set of ellipsoid nodes , and vector A ( u ) describes the configurations of each ellipsoid . The controller is a policy network parameterized by weights θ . The tuple formed by the graph and the policy is defined as a species , denoted as Ω = ( G , θ ) . Generation and Policy Sharing . In the j-th iteration , NGE evaluates a pool of species called a generation , denoted as P j = { ( Gji , θ j i ) , ∀i = 1 , 2 , ... , N } , where N is the size of the generation . In NGE , the search space includes not only the graph space , but also the weight or parameter space of the policy network . For better efficiency of NGE , we design a process called Policy Sharing ( PS ) , where weights are reused from parent to child species . The details of PS is described in Section 3.4 . Our model can be summarized as follows . NGE performs population-based optimization by iterating among mutation , evaluation and selection . The objective and performance metric of NGE are introduced in Section 3.1 . In NGE , we randomly initialize the generation with N species . For each generation , NGE trains each species and evaluates their fitness separately , the policy of which is described in Section 3.2 . During the selection , we eliminate K species with the worst fitness . To mutate K new species from surviving species , we develop a novel mutation scheme called Graph Mutation with Uncertainty ( GM-UC ) , described in Section 3.3 , and efficiently inherit policies from the parent species by Policy Sharing , described in Section 3.4 . Our method is outlined in Algorithm 1 .

Summary: This paper tackles the problem of automatic robot design. The most popular approach to doing this has been evolutionary methods which work by evolving morphology of agents in a feed-forward manner using a propagation and mutation rules. This is a non-differentiable process and relies on maintaining a large pool of candidates out of which best ones are chosen with the highest fitness. In robot design for a given task using rewards, training each robot design using RL with rewards is an expensive process and not scalable. This paper uses graph network to train each morphology using RL. Thereby, allowing the controller to share parameters and reuse information across generations. This expedites the score function evaluation improving the time complexity of the evolutionary process.