Evolution Gym: A Large-Scale Benchmark for Evolving Soft Robots

1 Introduction . One of the main goals of artificial intelligence is to develop effective approaches for the creation of embodied intelligent systems . Inspired from real organisms , where body structure and brain are two key factors for completing any task in a real environment , a successful intelligent robot typically requires concurrently optimizing its structure design and control mechanism . Such a co-design problem has been a long-standing key challenge in the robotics and machine learning communities . Surprisingly , despite its importance , most previous research works still either only develop complex control algorithms for existing robot structures [ 1 , 2 , 17 , 30 ] , or conduct co-optimization over robot morphology and control for only a few simple tasks ( e.g. , running , jumping ) [ 7 , 14 , 31 , 32 ] , especially 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . in the soft body domain . The primary reasons behind the under-exploration of co-design algorithms in sophisticated problems are : ( 1 ) the underlying complex bilevel optimization scheme of a co-design algorithm , where the inner control optimization loop leads to a long iteration cycle of the whole optimization process ; ( 2 ) the lack of a well-established benchmark platform providing the researchers with a suite to evaluate and compare different algorithms . Digital benchmark environments have proven to be successful at promoting the development of advanced learning techniques via providing a comprehensive evaluation suite to make fair comparisons among different algorithms [ 5 , 11 , 36 ] . However , to our best knowledge , all existing benchmark platforms constrain their domains within control optimization problems , and the space of co-optimization environment suites is still rarely explored . To fill this gap , in this work we propose Evolution Gym , a large-scale benchmark for evolving both the shape structure and controller of soft robots . The body of each robot in Evolution Gym is composed of various types of primitive building blocks ( e.g. , soft voxels , rigid voxels , actuator voxels ) , and the control of the robot includes action signals applied on the actuator voxels . We choose to use this multi-material voxel-based structure as the representation of robot body since it provides a general and universal representation for various categories of robot designs , and at the same time results in a modular and expressive structure design space . We adopt a mass-spring dynamics system [ 26 ] with penalty-based frictional contact as the underlining physics engine . Such a light-weight simulator allows the co-design algorithms to significantly reduce the simulation cost and thus accelerate the develop-evaluate iteration cycle [ 3 , 15 , 23 ] . The back-end simulator is fully developed in C++ to provide further computing efficiency . Another feature of Evolution Gym is its large variety of tasks categorized by varying difficulty levels , which offer an extensive evaluation benchmark for comparing approaches . The benchmark is currently comprised of more than 30 tasks , spanning locomotion on various types of terrains and manipulation . Moreover , Evolution Gym is easy to use . In order to have user-friendly interfaces , we build a Python wrapper outside the C++ simulator and carefully design our APIs off of the well-received APIs of OpenAI Gym with minimum modifications . Evolution Gym will be released fully open-source under the MIT license . In addition , we develop several baseline algorithms by integrating state-of-the-art design optimization approaches and reinforcement learning techniques . Specifically , in our baseline algorithms , design optimization methods are served in the outer loop to evolve the physical structures of robots and reinforcement learning algorithms are applied in the inner loop to optimize a controller for a given proposed structure design . We conduct extensive experiments to evaluate all baseline algorithms on Evolution Gym . The experiment results demonstrate that intelligent robot designs can be evolved fully autonomously while outperforming hand-designed robots in easier tasks , which reaffirms the necessity of jointly optimizing for both robot structure and control . However , none of the baseline algorithms are capable enough to successfully find robots that complete the task in our hardest environments . Such insufficiency of the existing algorithms suggests the demand for more advanced robot co-design techniques , and we believe our proposed Evolution Gym provides a comprehensive evaluation testbed for robot co-design and unlocks future research in this direction . In summary , our work has the following key contributions : ( i ) We propose Evolution Gym , the first large-scale benchmark for soft robot co-design algorithms . ( ii ) We develop several co-design algorithms by combining state-of-the-art design optimization methods and deep reinforcement learning techniques for control optimization . ( iii ) The developed algorithms are evaluated and analyzed on our proposed benchmark suite , and the results validate the efficacy of robot co-design while pointing out the failure and limitations of existing algorithms . 2 Related work . Robot co-design Co-designing the structure ( i.e. , body ) and control ( i.e. , brain ) of robots is a long-standing key challenge in the robotics community . As the earliest work in this space , Sims [ 31 ] represents the structure of a rigid robot as a directed graph and proposes an evolutionary algorithm defined on graphs to optimize the robot design . Subsequently , the co-design of rigid robots is formulated as a graph search problem where more efficient search algorithms are applied [ 13 , 27 , 39 , 41 ] to achieve increasingly interesting results . However , with the restriction of having rigid components only , these algorithms are unable to produce optimal or even feasible designs for many challenging tasks where a compliant joint or robot component is required to achieve the goal . On the contrary , soft components offer much more flexibility to represent arbitrary shapes , making the design of more complex , agile , and high-performing robots possible . Inspired by this , some work has been conducted to co-design robots composed of soft cells . Cheney et al . [ 7 , 8 ] ; Van Diepen and Shea [ 37 ] ; Corucci et al . [ 10 ] propose evolutionary algorithms to co-optimize the structure and control of voxel-based robots . However those algorithms typically parameterize the control as an open-loop periodic sequence of actuation , which prevents robots from learning complex non-periodic tasks such as walking on uneven or varying terrains . Spielberg et al . [ 32 ] and Medvet et al . [ 23 ] jointly optimize the spatial-varying material parameters and the neural network policy for soft robots but leave the shape of the robot fixed . Our proposed benchmark shares a similar expressive structure design space as Cheney et al . [ 7 ] , but allows the control to be parameterized by a sophisticated neural network feedback policy . To handle such sophisticated joint optimization of the robot structure and high-dimensional neural network control policy , we develop several baseline co-design algorithms by combining state-of-the-art design optimization strategies and reinforcement learning techniques for control optmization . Benchmark environments for robotics learning Present research in robotics learning is largely facilitated by emerging benchmark environments . For instance , OpenAI Gym [ 5 ] , DeepMind Control Suite [ 36 ] , rllab [ 11 ] , and Gibson [ 40 ] have been developed to benchmark RL algorithms for controlling rigid robots . At the same time , PlasticineLab [ 16 ] is specifically designed for soft robot learning . However , the existing benchmark environments are all constructed for learning the control only . To enable the possibility of evolving the structure of a robot , the existing co-design work has to either implement their own testing environment [ 32 , 7 , 8 , 10 , 37 ] , or make substantial changes on the underlying code of the existing control-only environments [ 29 ] . The independent development of testing beds requires non-trivial workload , and as a result , existing co-design works mainly focus on evaluating the robot on a few simple tasks such as walking on a flat terrain [ 7 , 6 , 8 , 37 , 32 , 23 ] , or swimming along a single direction [ 9 , 39 ] . An unintended consequence of such independency is an indirect comparison among different algorithms . Evolution Gym fills this gap by presenting a large variety of tasks with different difficulty levels that span from locomotion to manipulation . The proposed benchmark suite can be effectively used to test the generalizability of the algorithms on different tasks , potentially accelerating research in robot co-design . 3 Evolution Gym 3.1 Overview . In this section , we present Evolution Gym , a large-scale benchmark for the co-design of voxel-based soft robots . Evolution Gym is featured by its versatile and expressive multi-material voxel-based structure design space , flexibility of the controller parameterization , wide spectrum of tasks of various difficulty levels , fast back-end soft-body simulation support , and user-friendly Python interfaces . As shown in the overview in Figure 1 , Evolution Gym is comprised of a task-specific environment and a back-end soft-body simulator . The gym suite provides seamless interfaces with a user-defined co-design algorithm . The co-design algorithm typically consists of a design optimizer and a control optimizer . The design optimizer can propose a new robot structure to the control optimizer , then the control optimizer will compute an optimized controller for the given structure through interactions with Evolution Gym and finally return the maximum reward that this robot structure can achieve . In this way , Evolution Gym provides an easy-to-use platform for co-design algorithms to evolve both robot structure and control to optimize for robots ’ task performances . Evolution Gym is designed to be the first comprehensive testbed for benchmarking and comparing different co-design algorithms with the hope to facilitate the development of more novel and powerful algorithms in the co-design field . 3.2 Multi-material voxel-based representation . Evolution Gym employs a unified multi-material voxel-based representation for all the components in the environment ( e.g. , robot , terrain , object ) as shown in Figure 1A . Specifically , each robot in our gym is composed of rigid voxels , soft voxels , horizontal/vertical actuator voxels , and empty voxels . For terrain and objects , we use the same voxel-based structure but with passive voxel types ( i.e. , soft/rigid voxels ) . We chose a voxel-based representation for three main reasons . First , such a multi-material structure of robots provides a general and universal representation for various categories of robot designs and results in a modular structure design space . Additionally , with just the few voxel types described above , and less than 100 voxels per robot , we are able to construct a wide diversity of morphologies due to the resulting combinatorial robot design space . Even with this simple representation , our designed robots are capable of performing complex motions and completing difficult tasks . Finally , voxel-based robots can be simulated by a fast mass-spring simulation ( see section 3.4 ) which allows our framework to be efficient enough to train robots in a matter of minutes and provides a computationally tractable benchmark for iterating co-design algorithms . 3.3 Task representation . Each task in Evolution Gym contains a robot structure proposed by the co-design algorithm , environment specifications ( e.g. , terrain , object ) , and a task-related goal ( e.g. , locomotion or manipulation ) . The tasks interface with the co-design algorithm through a few key elements including robot structure specification , observation , action , and reward . We introduce each element in detail below . Robot structure specification As described in Section 3.2 , we construct each robot from primitive building blocks arranged on a grid layout . In code , each robot is specified as a material matrix of voxelsM and a connection link list C. The value of entry m ∈M is a label corresponding to a voxel type from the set { Empty , Rigid , Soft , Horizontal Actuator , Vertical Actuator } . The connection link list C stores a list of connection pairs of adjacent voxels . The co-design algorithm can update the robot structure in the environment through initialization function withM and C as arguments . Observation The observation is composed in each step to inform the controller of state information of the robot , terrain information of the environment , and goal-relevant information . More specifically , let N be the total number of voxel corner points of the robot . Then the state information of the robot in our tasks is a ( 2N +3 ) -D vector including the relative position of each voxel corner with respect to the center of mass of the robot ( 2N -D ) , and the velocity and orientation of center of mass ( 3-D ) . To handle complex tasks , specifically those with varying terrain types , an additional observation vector including terrain information is provided . We compile terrain information within a local window of size 2W around the robot into a length-2W vector observation that describes the terrain ’ s elevation . Furthermore , goal-related information is offered to inform the controller of the execution status of the current task . This goal-related observation is task-specific and is defined on each task separately . For instance , in manipulation tasks where the robot interacts with some object O , we provide orientation and velocity as well as the position of O ’ s center of mass relative to the robot . Action At each time step , an action vector from the robot ’ s controller is provided to step Evolution Gym ’ s simulator . In Evolution Gym , each component of the action vector is associated with an actuator voxel ( either horizontal or vertical ) of the robot , and instructs a deformation target of that voxel . Specifically , the action value u is within the range [ 0.6 , 1.6 ] , and corresponds to a gradual expansion/contraction of that actuator to u times its rest length . Reward Each task is equipped with a reward function measuring the performance of the current robot and the control action . The value of the reward is defined step-wise and is fed back to the agent through step function . The reward function is highly task-specific and should be defined to precisely characterize the robot ’ s completeness of the task . Please refer to Section 3.5 and Appendix for detailed descriptions of the reward functions on each task .

Summary: The paper proposes a benchmark for Evolving Soft Robots. The benchmark is designed to facilitate research in co-design of control algorithms and robot morphologies. The proposed framework relies on a custom physics engine which allows a simple mass-spring-damper connection of individual modules. The proposed benchmark is suitable for both manipulation and locomotion even though restricted to 2D environments only. Authors provide also an initial evaluation of SoTA design optimization algorithms (genetic algorithm, bayesian optmisation, CPPN-NEAT) on the proposed benchmark.