Evolution Gym: A Large-Scale Benchmark for Evolving Soft Robots

1 Introduction . One of the main goals of artificial intelligence is to develop effective approaches for the creation of embodied intelligent systems . Inspired from real organisms , where body structure and brain are two key factors for completing any task in a real environment , a successful intelligent robot typically requires concurrently optimizing its structure design and control mechanism . Such a co-design problem has been a long-standing key challenge in the robotics and machine learning communities . Surprisingly , despite its importance , most previous research works still either only develop complex control algorithms for existing robot structures [ 1 , 2 , 17 , 30 ] , or conduct co-optimization over robot morphology and control for only a few simple tasks ( e.g. , running , jumping ) [ 7 , 14 , 31 , 32 ] , especially 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . in the soft body domain . The primary reasons behind the under-exploration of co-design algorithms in sophisticated problems are : ( 1 ) the underlying complex bilevel optimization scheme of a co-design algorithm , where the inner control optimization loop leads to a long iteration cycle of the whole optimization process ; ( 2 ) the lack of a well-established benchmark platform providing the researchers with a suite to evaluate and compare different algorithms . Digital benchmark environments have proven to be successful at promoting the development of advanced learning techniques via providing a comprehensive evaluation suite to make fair comparisons among different algorithms [ 5 , 11 , 36 ] . However , to our best knowledge , all existing benchmark platforms constrain their domains within control optimization problems , and the space of co-optimization environment suites is still rarely explored . To fill this gap , in this work we propose Evolution Gym , a large-scale benchmark for evolving both the shape structure and controller of soft robots . The body of each robot in Evolution Gym is composed of various types of primitive building blocks ( e.g. , soft voxels , rigid voxels , actuator voxels ) , and the control of the robot includes action signals applied on the actuator voxels . We choose to use this multi-material voxel-based structure as the representation of robot body since it provides a general and universal representation for various categories of robot designs , and at the same time results in a modular and expressive structure design space . We adopt a mass-spring dynamics system [ 26 ] with penalty-based frictional contact as the underlining physics engine . Such a light-weight simulator allows the co-design algorithms to significantly reduce the simulation cost and thus accelerate the develop-evaluate iteration cycle [ 3 , 15 , 23 ] . The back-end simulator is fully developed in C++ to provide further computing efficiency . Another feature of Evolution Gym is its large variety of tasks categorized by varying difficulty levels , which offer an extensive evaluation benchmark for comparing approaches . The benchmark is currently comprised of more than 30 tasks , spanning locomotion on various types of terrains and manipulation . Moreover , Evolution Gym is easy to use . In order to have user-friendly interfaces , we build a Python wrapper outside the C++ simulator and carefully design our APIs off of the well-received APIs of OpenAI Gym with minimum modifications . Evolution Gym will be released fully open-source under the MIT license . In addition , we develop several baseline algorithms by integrating state-of-the-art design optimization approaches and reinforcement learning techniques . Specifically , in our baseline algorithms , design optimization methods are served in the outer loop to evolve the physical structures of robots and reinforcement learning algorithms are applied in the inner loop to optimize a controller for a given proposed structure design . We conduct extensive experiments to evaluate all baseline algorithms on Evolution Gym . The experiment results demonstrate that intelligent robot designs can be evolved fully autonomously while outperforming hand-designed robots in easier tasks , which reaffirms the necessity of jointly optimizing for both robot structure and control . However , none of the baseline algorithms are capable enough to successfully find robots that complete the task in our hardest environments . Such insufficiency of the existing algorithms suggests the demand for more advanced robot co-design techniques , and we believe our proposed Evolution Gym provides a comprehensive evaluation testbed for robot co-design and unlocks future research in this direction . In summary , our work has the following key contributions : ( i ) We propose Evolution Gym , the first large-scale benchmark for soft robot co-design algorithms . ( ii ) We develop several co-design algorithms by combining state-of-the-art design optimization methods and deep reinforcement learning techniques for control optimization . ( iii ) The developed algorithms are evaluated and analyzed on our proposed benchmark suite , and the results validate the efficacy of robot co-design while pointing out the failure and limitations of existing algorithms . 2 Related work . Robot co-design Co-designing the structure ( i.e. , body ) and control ( i.e. , brain ) of robots is a long-standing key challenge in the robotics community . As the earliest work in this space , Sims [ 31 ] represents the structure of a rigid robot as a directed graph and proposes an evolutionary algorithm defined on graphs to optimize the robot design . Subsequently , the co-design of rigid robots is formulated as a graph search problem where more efficient search algorithms are applied [ 13 , 27 , 39 , 41 ] to achieve increasingly interesting results . However , with the restriction of having rigid components only , these algorithms are unable to produce optimal or even feasible designs for many challenging tasks where a compliant joint or robot component is required to achieve the goal . On the contrary , soft components offer much more flexibility to represent arbitrary shapes , making the design of more complex , agile , and high-performing robots possible . Inspired by this , some work has been conducted to co-design robots composed of soft cells . Cheney et al . [ 7 , 8 ] ; Van Diepen and Shea [ 37 ] ; Corucci et al . [ 10 ] propose evolutionary algorithms to co-optimize the structure and control of voxel-based robots . However those algorithms typically parameterize the control as an open-loop periodic sequence of actuation , which prevents robots from learning complex non-periodic tasks such as walking on uneven or varying terrains . Spielberg et al . [ 32 ] and Medvet et al . [ 23 ] jointly optimize the spatial-varying material parameters and the neural network policy for soft robots but leave the shape of the robot fixed . Our proposed benchmark shares a similar expressive structure design space as Cheney et al . [ 7 ] , but allows the control to be parameterized by a sophisticated neural network feedback policy . To handle such sophisticated joint optimization of the robot structure and high-dimensional neural network control policy , we develop several baseline co-design algorithms by combining state-of-the-art design optimization strategies and reinforcement learning techniques for control optmization . Benchmark environments for robotics learning Present research in robotics learning is largely facilitated by emerging benchmark environments . For instance , OpenAI Gym [ 5 ] , DeepMind Control Suite [ 36 ] , rllab [ 11 ] , and Gibson [ 40 ] have been developed to benchmark RL algorithms for controlling rigid robots . At the same time , PlasticineLab [ 16 ] is specifically designed for soft robot learning . However , the existing benchmark environments are all constructed for learning the control only . To enable the possibility of evolving the structure of a robot , the existing co-design work has to either implement their own testing environment [ 32 , 7 , 8 , 10 , 37 ] , or make substantial changes on the underlying code of the existing control-only environments [ 29 ] . The independent development of testing beds requires non-trivial workload , and as a result , existing co-design works mainly focus on evaluating the robot on a few simple tasks such as walking on a flat terrain [ 7 , 6 , 8 , 37 , 32 , 23 ] , or swimming along a single direction [ 9 , 39 ] . An unintended consequence of such independency is an indirect comparison among different algorithms . Evolution Gym fills this gap by presenting a large variety of tasks with different difficulty levels that span from locomotion to manipulation . The proposed benchmark suite can be effectively used to test the generalizability of the algorithms on different tasks , potentially accelerating research in robot co-design . 3 Evolution Gym 3.1 Overview . In this section , we present Evolution Gym , a large-scale benchmark for the co-design of voxel-based soft robots . Evolution Gym is featured by its versatile and expressive multi-material voxel-based structure design space , flexibility of the controller parameterization , wide spectrum of tasks of various difficulty levels , fast back-end soft-body simulation support , and user-friendly Python interfaces . As shown in the overview in Figure 1 , Evolution Gym is comprised of a task-specific environment and a back-end soft-body simulator . The gym suite provides seamless interfaces with a user-defined co-design algorithm . The co-design algorithm typically consists of a design optimizer and a control optimizer . The design optimizer can propose a new robot structure to the control optimizer , then the control optimizer will compute an optimized controller for the given structure through interactions with Evolution Gym and finally return the maximum reward that this robot structure can achieve . In this way , Evolution Gym provides an easy-to-use platform for co-design algorithms to evolve both robot structure and control to optimize for robots ’ task performances . Evolution Gym is designed to be the first comprehensive testbed for benchmarking and comparing different co-design algorithms with the hope to facilitate the development of more novel and powerful algorithms in the co-design field . 3.2 Multi-material voxel-based representation . Evolution Gym employs a unified multi-material voxel-based representation for all the components in the environment ( e.g. , robot , terrain , object ) as shown in Figure 1A . Specifically , each robot in our gym is composed of rigid voxels , soft voxels , horizontal/vertical actuator voxels , and empty voxels . For terrain and objects , we use the same voxel-based structure but with passive voxel types ( i.e. , soft/rigid voxels ) . We chose a voxel-based representation for three main reasons . First , such a multi-material structure of robots provides a general and universal representation for various categories of robot designs and results in a modular structure design space . Additionally , with just the few voxel types described above , and less than 100 voxels per robot , we are able to construct a wide diversity of morphologies due to the resulting combinatorial robot design space . Even with this simple representation , our designed robots are capable of performing complex motions and completing difficult tasks . Finally , voxel-based robots can be simulated by a fast mass-spring simulation ( see section 3.4 ) which allows our framework to be efficient enough to train robots in a matter of minutes and provides a computationally tractable benchmark for iterating co-design algorithms . 3.3 Task representation . Each task in Evolution Gym contains a robot structure proposed by the co-design algorithm , environment specifications ( e.g. , terrain , object ) , and a task-related goal ( e.g. , locomotion or manipulation ) . The tasks interface with the co-design algorithm through a few key elements including robot structure specification , observation , action , and reward . We introduce each element in detail below . Robot structure specification As described in Section 3.2 , we construct each robot from primitive building blocks arranged on a grid layout . In code , each robot is specified as a material matrix of voxelsM and a connection link list C. The value of entry m ∈M is a label corresponding to a voxel type from the set { Empty , Rigid , Soft , Horizontal Actuator , Vertical Actuator } . The connection link list C stores a list of connection pairs of adjacent voxels . The co-design algorithm can update the robot structure in the environment through initialization function withM and C as arguments . Observation The observation is composed in each step to inform the controller of state information of the robot , terrain information of the environment , and goal-relevant information . More specifically , let N be the total number of voxel corner points of the robot . Then the state information of the robot in our tasks is a ( 2N +3 ) -D vector including the relative position of each voxel corner with respect to the center of mass of the robot ( 2N -D ) , and the velocity and orientation of center of mass ( 3-D ) . To handle complex tasks , specifically those with varying terrain types , an additional observation vector including terrain information is provided . We compile terrain information within a local window of size 2W around the robot into a length-2W vector observation that describes the terrain ’ s elevation . Furthermore , goal-related information is offered to inform the controller of the execution status of the current task . This goal-related observation is task-specific and is defined on each task separately . For instance , in manipulation tasks where the robot interacts with some object O , we provide orientation and velocity as well as the position of O ’ s center of mass relative to the robot . Action At each time step , an action vector from the robot ’ s controller is provided to step Evolution Gym ’ s simulator . In Evolution Gym , each component of the action vector is associated with an actuator voxel ( either horizontal or vertical ) of the robot , and instructs a deformation target of that voxel . Specifically , the action value u is within the range [ 0.6 , 1.6 ] , and corresponds to a gradual expansion/contraction of that actuator to u times its rest length . Reward Each task is equipped with a reward function measuring the performance of the current robot and the control action . The value of the reward is defined step-wise and is fed back to the agent through step function . The reward function is highly task-specific and should be defined to precisely characterize the robot ’ s completeness of the task . Please refer to Section 3.5 and Appendix for detailed descriptions of the reward functions on each task .

Summary: The paper presents Evolution Gym, a benchmark for the evaluation of design and control co-optimization algorithms on simulated voxel-based soft robots. It contains a back-end soft-body simulator (written in C++ with bindings in Python), a set of 30 evaluation tasks of various types and difficulty levels, 3 design optimization algorithms and 1 control optimization algorithm. The algorithms are evaluated on the benchmark tasks and the results are reported in the paper.