Evolution Gym: A Large-Scale Benchmark for Evolving Soft Robots
Authors: Jagdeep Singh Bhatia
Holly Jackson
Yunsheng Tian
Jie Xu
Wojciech Matusik
Venue: NeurIPS 2021 Poster (2021)
Keywords: Co-design
Benchmark
Soft Robotics
Evolutionary Algorithms
Reinforcement Learning
Machine Learning

Abstract: Both the design and control of a robot play equally important roles in its task performance. However, while optimal control is well studied in the machine learning and robotics community, less attention is placed on finding the optimal robot design. This is mainly because co-optimizing design and control in robotics is characterized as a challenging problem, and more importantly, a comprehensive evaluation benchmark for co-optimization does not exist. In this paper, we propose Evolution Gym, the first large-scale benchmark for co-optimizing the design and control of soft robots. In our benchmark, each robot is composed of different types of voxels (e.g., soft, rigid, actuators), resulting in a modular and expressive robot design space. Our benchmark environments span a wide range of tasks, including locomotion on various types of terrains and manipulation. Furthermore, we develop several robot co-evolution algorithms by combining state-of-the-art design optimization methods and deep reinforcement learning techniques. Evaluating the algorithms on our benchmark platform, we observe robots exhibiting increasingly complex behaviors as evolution progresses, with the best evolved designs solving many of our proposed tasks. Additionally, even though robot designs are evolved autonomously from scratch without prior knowledge, they often grow to resemble existing natural creatures while outperforming hand-designed robots. Nevertheless, all tested algorithms fail to find robots that succeed in our hardest environments. This suggests that more advanced algorithms are required to explore the high-dimensional design space and evolve increasingly intelligent robots -- an area of research in which we hope Evolution Gym will accelerate progress. Our website with code, environments, documentation, and tutorials is available at http://evogym.csail.mit.edu/.

Evolution Gym: A Large-Scale Benchmark for Evolving Soft Robots
Jagdeep Singh Bhatia
Holly Jackson MIT CSAIL
Yunsheng Tian MIT CSAIL
MIT CSAIL j jagdeep@mit.co edu
hjackson@mit..co edu
yunsheng@csail mit. edu
Jie Xu
Wojciech Matusik
MIT CSAIL
MIT CSAIL
jiex0csail wwt. edu
wojciechhceail, mit. edu
Abstract
Both the design and control of a robot play equally important roles in its task performance. However, while optimal control is well studied in the machine learn- ing and robotics community, less attention is placed on finding the optimal robot design. This is mainly because co-optimizing design and control in roboties is characterized as challenging problem, and more importantly, comprehensive evaluation benchmark for co-optimization does exist. In this paper, we propose Evolution Gym, the first large-scale benchmark for co-optimizing the design and control of soft robots. types of voxels (e.g., soft, rigid, actuators), resulting in modular and expressive robot design space. Our benchmark environments span wide range of tasks, including locomotion on various types of terrains and manipulation. Furthermore, we develop several robot co-evolution algorithms by combining state-of-the-art design optimization methods and deep reinforcement learning techniques. Eval- uating the algorithms on our benchmark platform, we observe robots exhibiting increasingly complex behaviors evolution progresses, with the best evolved designs solving many our proposed tasks. Additionally, even though robot de- signs are evolved autonomously from scratch without prior knowledge, they often grow to resemble existing natural creatures while outperforming hand-designed robots. Nevertheless, all tested algorithms fail to find robots that succeed in hardest environments. This suggests that more advanced algorithms are required explore the high-dimensional design space and evolve increasingly intelligent robots an area of research in which we hope Evolution Gym will accelerate progress. Our website with code, environments, documentation, and tutorials is available at http: / //v/g//vo.mmm///////o//////..... csail. mit edu.
our benchmark. each robot is composed of different
Introduction
One of the main goals of artificial intelligence to develop effective approaches for the creation of embodied intelligent systems. Inspired from real organisms, where body structure and brain are two key factors for completing any task real environment, a successful intelligent robot typically requires concurrently optimizing its structure design and control mechanism. Such co-design problem has been a long-standing key challenge in the robotics and machine learning communitics. Surprisingly, despite its importance, most previous research works still either only develop complex control algorithms for existing robot structures [1, 2, 17, 30], or conduct co-optimization over robot morphology and control for only few simple tasks (e.g., running, jumping) [7, 14, 31, 32), especially
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

in the soft body domain. The primary reasons behind the under- exdloreeploottion of co-design algorithms in sophisticated problems are: (1) the underlying complex bilevel optimization scheme of a co-design algorithm, where the inner control optimization loop leads to long iteration cycle of the whole optimization process; (2) the lack of well-established benchmark platform providing the researchers with suite to evaluate and compare different algorithms.
Digital benchmark environments have proven be successful at promoting the development of ad- vanced learning techniques via providing a comprehensive evaluation suite to make fair comparisons among different algorithms [5. 11, 36). However, to our best knowledge, all existing benchmark plat- forms constrain their domains within control optimization problems, and the space of co-optimization environment suites is still rarely explored.
To fill this gap, in this work we propose Evolution Gym, a large-scale benchmark for evolving both the shape structure and controller of soft robots. The body of each robot in Evolution Gym is composed of various types of primitive building blocks (e.g., soft voxels, rigid voxels, actuator voxels), and the control of the robot includes action signals applied on the actuator voxels. We choose to use this multi-material voxel-based structure as the representation of robot body since it provides a general and universal representation for various categories of robot designs, and at the same time results in a modular and expressive structure design space. We adopt a mass-spring dynamics system [26] with penalty- -based frictional contact as the underlining physies engine. Such light-weight simulator allows the co-design algorithms to significantly reduce the simulation cost and thus accelerate the develop-evaluae evaluate iteration cycle [3, 15, 23]. The back-end simulator is fully developed in C++ to provide further computing efficiency. Another feature of Evolution Gym is its large variety of tasks categorized by varying difficulty levels, which offer an extensive evaluation benchmark for comparing approaches. The benchmark is currently comprised of more than 30 tasks, spanning locomotion on various types of terrains and manipulation. Moreover, Evolution Gym is easy to use. In order to have user-friendly interfaces, we build a Python wrapper outside the C++ simulator and carefully design our APIs off of the well-received APIs of OpenAl Gym with minimum modifications. Evolution Gym will be released fully open-source under the MIT license.
In addition, we develop several baseline algorithms by integrating state-of of-the-art design optimization approaches and reinforcement learning techniques. Specifically, in our baseline algorithms, design optimization methods are served in the outer loop to evolve the physical structures of robots and reinforcement learning algorithms are applied in the inner loop to optimize controller for given proposed structure design. We conduct extensive experiments to evaluate all baseline algorithms on Evolution Gym. The experiment results demonstrate that intelligent robot designs can be evolved fully autonomously while outperforming hand-designed robots in easier tasks, which reaffirms the necessity of jointly optimizing for both robot structure and control. However, none of the baseline algorithms are capable enough to successfully find robots that complete the task in our hardest environments. Such insufficiency of the existing algorithms suggests the demand for more advanced robot co-design techniques, and we believe our proposed Evolution Gym provides a comprehensive evaluation testbed for robot co-design and unlocks future research in this direction.
In summary, our work has the following key contributions: (i) We propose Evolution Gym, the first large-scale benchmark for soft robot co-design algorithms. (ii) We develop several co-design algorithms by combining state-of-the-art design optimization methods and deep reinforcement learning techniques for control optimization. (iii) The developed algorithms are evaluated and analyzed on our proposed benchmark suite, and the results validate the efficacy of robot co-design while pointing out the failure and limitations of existing algorithms.
2
Related work
Robot co-design Co-designing the structure (i.e., body) and control (i.e., brain) of robots is long-standing key challenge in the robotics community. As the earliest work in this space, Sims [31] represents the structure of rigid robot as directed graph and proposes an evolutionary algorithm defined on graphs to optimize the robot design. Subsequently, the co-design of rigid robots formulated as a graph search problem where more efficient search algorithms are applied [13, 27, 39, 41] to achieve increasingly interesting results. However, with the restriction of having rigid components only, these algorithms are unable to produce optimal or even feasible designs for many challenging tasks where a compliant joint or robot component is required to achieve the goal.

---

On the contrary, soft components offer much more flexibility to represent arbitrary shapes, making the design of more complex, agile, and high-perttee performirn robots possible. Inspired by this, some work has been conducted to co-design robots composed of soft cells. Cheney et al. [7, 8]; Van Diepen and Shea [37]; Corucci et al. [10] propose evolutionary algorithms co-optimize the structure and control of voxel-based robots. However those algorithms typically parameterize the control as an open-loop periodic sequence of actuation, which prevents robots from learning complex non-periodic tasks such walking uneven or varying terrains. Spielberg et al. [32] and Medvet et al. [23] jointly optimize the spatial-varying material parameters and the neural network policy for soft robots but leave the shape of the robot fixed. Our proposed benchmark shares similar expressive structure design space as Cheney et al. [7], but allows the control to be parameterized by a sophisticated neural network feedback policy. To handle such sophisticated joint optimization of the robot structure and high-dimensional neural network control policy, we develop several baseline co-design algorithms by combining state-of-the-ar design optimization strategies and reinforcement learning techniques for control optmization.
Benchmark environments for robotics learning Present research in robotics learning is largely facilitated by emerging benchmark environments. For instance, OpenAl Gym [5], DeepMind Control Suite [36), rllab [11], and Gibson [40] have been developed to benchmark RL algorithms for controlling rigid robots. At the same time, PlasticineLab [16] is specifically designed for soft robot learning. However, the existing benchmark environments only. To enable the possibility of evolving the structure robot, the existing co-design work has to either implement their own testing environment [32, 7, 8. 10, 37J. make substantial changes the underlying code of the existing control-only environments [29]. The independent development of testing beds requires non-trivial workload, and as result, existing co-design works mainly focus on evaluating the robot on a few simple tasks such as walking on a flat terrain [7, 6, 8, 37, 32, 23], or swimming along a single direction [9, 39J. An unintended consequence of such independency is an indirect comparison among different algorithms. Evolution Gym fills this gap by presenting large variety of tasks with different difficulty levels that span from locomotion manipulation. The proposed benchmark suite can be effectively used to test the generalizability of the algorithms on different tasks, potentially accelerating research in robot co-design.
all constructed for learning the control
3 Evolution Gym
Evolution Gym
Co-Design Algorithm
Back-End Soft Body Simulator
Design Optimization
Actuator
New Design
Task Specific Environments
Observation
Reward
Control Optimization
Action
Design
Evolution Gym
Figure I: Overview of Evolution Gym and its integration with the co-design algorithms. Evolu- tion Gym comprised of a back-end soft body simulator (A, B) and task-specific environments (C). user-customized co-design algorithm can be plugged to optimize for both robot structure and control through interacting with Evolution Gym certain task.
3.1 Overview
In this section, we present Evolution Gym, a large-scale benchmark for the co-design of voxel-based soft robots. Evolution Gym is featured by its versatile and expressive multi-material voxel-based structure design space, flexibility of the controller parameterization, wide spectrum of tasks of various difficulty levels, fast back-end soft-body simulation support, and user-friendly Python interfaces.

---

As shown in the overview in Figure 1, Evolution Gym is comprised of a task-specific environment and back-end soft-body simulator. The gym suite provides seamless interfaces with a user-defined co-design algorithm. The co-design algorithm typically consists of a design optimizer and a control optimizer. The design optimizer can propose new robot structure to the control optimizer, then the control optimizer will compute an optimized controller for the given structure through interactions with Evolution Gym and finally return the maximum reward that this robot structure can achieve. In this way, Evolution Gym provides an easy-to-use platform for co-design algorithms to evolve both robot structure and control to optimize for robots' task performances. Evolution Gym is designed to be the first comprehensive testbed for benchmarking and comparing different co-design algorithms with the hope to facilitate the development of more novel and powerful algorithms in the co-design field.
3.2 Multi-material voxel-based representation
Evolution Gym employs unified multi-materinn voxel-based representation for all the components in the environment (e.g., robot, terrain, object) as shown in Figure 1A. Specifically, each robot in our gym is composed of rigid voxels, soft voxels, horizontal/vertical actuator voxels, and empty voxels. For terrain and objects, we use the same voxel-based structure but with passive voxel types (i.e,, soft/rigid voxels).
We chose a voxel-based representation for three main reasons. First, such multi- ciltiiimtteraal structure of robots provides a general and universal representation for various categories of robot designs and results in a modular structure design space. Additionally, with just the few voxel types described above, and less than 100 voxels per robot, we are able to construct a wide diversity of morphologies due the resulting combinatorial robot design space. Even with this simple representation. our designed robots are capable of performing complex motions and completing difficult tasks. Finally, voxel-based robots can be simulated by fast mass-spring simulation (see section 3.4) which allows our framework to be efficient enough to train robots in matter of minutes and provides a computationally tractable benchmark for iterating co-design algorithms.
3.3
Task representation
Each task in Evolution Gym contains robot structure proposed by the co-design algorithm, environ- ment specifications (e.g.. terrain, object), and a task-related goal (e.g., locomotion or manipulation). The tasks interface with the co-design algorithm through a few key elements including robot structure specification, observation, action, and reward. We introduce each element in detail below.
Robot structure specification As described in Section 3.2, we construct each robot from primitive building blocks arranged on a grid layout. In code, each robot is specified as a material matrix of voxels M and a connection link list C. The value of entry type from the set (Empty, Rigid, Soft, Horizontal Actuator, Vertical Actuator). The connection link list C stores list of connection pairs of adjacent voxels. The co-design algorithm robot structure in the environment through initialization function with M and C as arguments.
.M is a label corresponding to a voxel
update the
The observation is composed in each step to inform the controller of state information of the robot, terrain information of the environment, and goal-relevant information. More specifically, let N be the total number of voxel corner points of the robot. Then the state information of the robot in our tasks is a (2NV + 3-D vector including the relative position of each voxel corner with respect to the center of mass of the robot (2N-D), and the velocity and orientation of center of mass (3-D). To handle complex tasks, specifically those with varying terrain types, an additional observation vector including terrain information is provided. We compile terrain information within a local window of size 2W around the robot into a length-22 22W vector observation that describes the terrain's 's clevation. Furthermore, goal-related information is offered to inform the controller of the execution status of the current task. This goal-related observation is task-specific and is defined on each task separately. For instance, in manipulation tasks where the robot interacts with some object O provide orientation and velocity as well as the position of O's center of mass relative to the robot.
Observation
Action At each time step, an action vector from the robot's controller is provided to step Evolution Gym's simulator. In Evolution Gym, each component of the action vector is associated with an actuator voxel (either horizontal or vertical) of the robot, and instructs deformation target of that

---

Locomotion Tasks
Object Manipulation Tasks
Walker [Easy)
(Easy)
Catcher (Hard)
Bridge Walker (Easy)
Thrower (Medium)
Climber Mottduum
Up Stepper (Medium)
Beam Slider (Hard)
Lifter (Hard)
Traverser (Hard)
Figure 2: visual overview selected 10 environments from Evolution Gym. A verbal descrip- tion of tasks is provided in Section 3.5.
voxel. Specifically, the action value within the range [0.6, 1.6], and corresponds to a gradual expansion/contraation of that actuator to u times its rest length.
Reward Each task is equipped with reward function measuring the performance of the current robot and the control action. The value of the reward is defined step-wise and fed back to the agent through step function. The reward function highly task-specific and should be defined to precisely characterize the robot's s completeness of the task. Please refer to Section 3.5 and Appendix for detailed descriptions of the reward functions on each task.
3.4 Simulation engine
We model the dynamics of the underlying simulator as 2D mass-spring system [26]. This simple, flexible formulation allows us to efficiently model soft robots with a wide range of capabilities in wide range of environments. The simulation engine is written entirely in C++. We create Python bindings of our simulator so seamlessly interfaces with standard learning frameworks.
The simulation represents objects and their environment as a mass-spring system in grid-like layout (Figure IB). Objects and their environments are initialized voxels. On initialization, each voxel is a cross cross-braced square, but may undergo deformation as the simulation progresses. Each edge acts as an ideal spring obeying Hooke's 's law, with spring constant defined by one of five possible material types. We employ symplectic RK-4 integration to step forward the simulation.
of non-overlapping, connected
Collision detection performed using a bounding-box tree structure [12]. Penalty-based contact forces and frictional forces computed proportionally to the depth of penetration of the correspond- ing voxels in contact, and are applied on the voxel vertices in the normal and tangential directions of the contact respectively. Please refer Appendix A for more details of simulation.
3.5 Benchmark environment suite
We have developed over 30 unique tasks with Evolution Gym and select tasks here to illustrate the diversity and comprehensiveness of our benchmark task set. All tasks are organized into two categories locomotion and manipulation though some tasks are mix of both. We further classify the tasks into different difficulty levels (i.e., easy, medium, hard) based on the performance of the baseline algorithms (see Section on them. We briefly introduce the selected tasks in this section. For more detailed descriptions and visualizations of the tasks, please refer to our website or Appendix B. It is also worth mentioning that our gym designed be extendable and the user can easily create new tasks for their needs.
3.5.1 Locomotion tasks
Walker (Fasy) This a common standard task typically considered by previous works where the robot needs to walk on flat terrain as fast as possible.

---

Bridge Walker (Easy) In this task, the robot traverses series of soft "rope" bridges separated by fixed pillars, and similarly as before needs maximize forward speed.
Up Stepper (Medium) The agent walks up fixed staircase with steps of varying length.
Climber (Medium) The robot must climb two tall fixed walls on each side. The robot is rewarded by its upward climbing speed.
Traverser (Hard) In this hard task, the robot needs to traverse a pit of rigid blocks to get to the other side without sinking into the pit.
3.5.2 Object manipulation tasks
Carrier (Easy) The robot needs to catch small, soft rectangular object initially dropped from above and then carry it along the forward direction. The robot rewarded by the distance both it and the object have traveled.
Thrower (Medium) The rohot throws soft rectangular box far possible without moving itself significantly from its original position.
Beam Slider (Hard) In this task, beam sits on top of a set of spaced-out floating platforms. The robot is rewarded for moving to the beam and sliding the forward direction.
Catcher (Hard) The agent needs to catch a spinning object randomly falling from a high location.
Lifter (Hard) The robot has manipulate an object and lift it out of hole.
4 Evolving soft robots
Robot evolution/co-destt algorithms are formulated as two-level optimization problem, which involves a design optimization method that evolves physical structures of the robots in the outer loop and control optimization algorithm that computes an optimized controller for a given robot structure the inner loop, as illustrated Algorithm 1. We briefly introduce several instantiations of design optimization methods and control optimization methods in Section 4.1 and 4.2 that we use for evaluation on our benchmark, and more details can be found in Appendix C.
Algorithm 1 Algorithmic framework of robot evolution
Inputs: Task specification T, number generations n, population size p.
Outputs: The best robot design D" and controller C*. S+
// Dataset of robot designs, controllers and reward // Sample initial population of robot designs
D1. ... Dp +- SAMPLEDESIGNS(p))
for i <- to n do
for
1 to p do
Cj <- OPTIMIZECONTROLA Dj) // Optimize the controller of given robot design r. EVALUATEREWARR([? Dú. Cj) Evaluate the reward of given design and controller // Update the evaluation result to the dataset
S + S S ((D,,,, r,,r,)} Di,..., D, + OPTIMIZEDESICNS(S, p) // Optimize a population of robot designs to evaluate Find the best design D and controller C* in dataset S with the maximum reward r".
4.1 Design optimization
Design optimization aims at evolving robot structures to maximize the reward under two physical constraints: the body has to be connected, and actuators must exist. In this section, we introduce three instantiations of the design optimization algorithm (OPTIMIZEDESIGN in Algorithm 1).
Genetic algorithm (GA) GAs [24] are widely used in optimizing black-box functions by relying on biologically inspired operators such as mutation, crossover and selection, demonstrated in previous works on evolving rigid robots [31, 39]. We implement simple GA using elitism selection and a each generation,
simple mutation strategy to evolve the population of robot designs. Specifically, our elitism selection works by keeping the top x% of the robots from the current population as survivors and discarding the rest, where x decreases gradually from 60 to 0 over generations. Next, we iteratively sample and mutate one of those survivors with 10% probability of changing each voxel

---

of the robot to create more offssringss Note that by mutating voxel type from/to empty voxel, we are able to change the topology of the robot. The crossover operator not implemented in our genetic algorithm.
Bayesian optimization (BO) BO (20, 25] is commonly used global optimization method for black-box functions by learning and utilizing a surrogate model, which is usually employed to optimize expensive-t--evaluate functions, including evolving rigid robots in previous works [29, 21]. Specifically, we choose a batch BO algorithm as described in Kandasamy et al. [18] and implemented in the GPyOpt package [4] that supports categorical input data. We use Gaussian processes as the surrogate model, batch Thompson sampling for extracting the acquisition function, and L-BFGS algorithm to optimize the acquisition function. To ensure fair comparison with other population- based evolutionary baseline algorithms, the batch size of this algorithm is set equal to the population size of other algorithms.
CPPN-NEAT CPPN-NEAT is the predominant method for evolving soft robot design in previous literature [6. 7, 8J. In this method, the robot design parameterized by a Compositional Pattern Producing Network (CPPN) [33]. The input to a CPPN is the spatial coordinate of a robot voxel and the output is the type of that voxel. Therefore, by querying the CPPN at all the spatial locations of a robot, we can obtain the type for each voxel to construct a robot. At the same time the NeuroEvolution of Augmenting Topologies (NEAT) algorithm [34] is used to evolve the structure of CPPNs by working as genetic algorithm with specific mutation, crossover, and selection operators defined on network structures. Our implementation of CPPN-NEAT based on the PyTorch-NEAT library [28) and the neat-python library [22].
4.2 Control optimization
In this section, we introduce the specific control optimization algorithm (OPTIMIZECONTRO) in Algorithm 1) that we use in the robot evolution algorithms. In previous works on evolving soft robots, the controller either encoded as fixed periodic sequence of actuation [7] or is parameterized as CPPN that outputs the frequency and phase offset of the periodic actuation for each voxel [8]. However, the periodic pattern of the control prevents robots from learning complex non-periodic tasks such as walking uneven varying terrains. Therefore, [35] to train the controller, making it possible for the soft robots to perform arbitrarily complex tasks in our benchmark. Specifically, we apply tfate-of-the-art RL algorithm named Proximal Policy Optimization (PPO) [30] for control optimization of robots, with code implementation given by [19].
use reinforcement learning (RL)
Experiments and results
In this section we present the evaluation results of baseline robot co-design algorithms on 10 selected benchmark tasks described in Section 3.5. The complete evaluation results on all our benchmark tasks can found Appendix E.
We develop three baseline algorithms for robot evolution by combing the three design optimization methods in Section 1.1 and PPO for control optimization in Section 4.2. Since the control optimization method is the same for all baseline algorithms, we simply use GA, BO, CPPN-NEAT to denote these three baseline algorithms with different design optimization methods. The evaluations of our baseline algorithms are performed on machines with Intel Xeon CPU @ 2.80GHz* 80 processors on Google Cloud Platform; GPU is not required. Evaluating one algorithm on single task usually takes several hours to twenty hours, depending on the number of evaluations, size of population, etc. See Appendix D for more details on hyperparameters of all the experiments.
5.1 Comparisons among baseline algorithms
We plot the reward curves of the three baseline algorithms on 10 selected benchmark tasks in Figure 3. There no single optimal algorithm that performs the best on all tasks, but overall, GA outperforms the other two baseline algorithms. This is surprising because our genetic algorithm is implemented with simple and intuitive operators for mutation and selection without sophisticated mechanisms. Therefore, we believe that with more carefully designed operators, GA has the potential to evolve much more intelligent robots. CPPN-NEAT generally performs well on locomotion tasks, as tested by previous works, but performs poorly more complex manipulation tasks. This is possibly because
7

---

NEAT favors CPPNs with simpler structures, which encourages CPPNs to generate robots with more regular patterns. However, succeed in complex manipulation tasks, some agile substructures of the robot must evolve, which might only exist in robots with irregular patterns. Finally, it is not surprising that BO performs poorly on most of the tasks because high-dimensional categorical input parameter space and the noisy evaluation done by RL together pose challenge to fitting an accurate surrogate model in BO.
Walker-v0
BridgeWalker-6
Climber-v0
UpStepper-vO
Traverser-vo
10.5
6
Reward
9.5
100
400
400
Carrier-v0
Thrower-v0
BeamSlider-vd
Catcher-vO
Lifter-v0
3.0
1.0
Reward
2.5
-1.5
2.0
-3.0
600 Evaluations
200 Evaluations
Evaluations
Evaluations
Evalustions
GA
CEAN
BO
Figure 3: Performance comparison among baseline algorithms. We plot the best performance of robots that each algorithm has evolved w.r.t. the number of evaluations on each task. All the curves averaged over 6 different random seeds, and the variance is shown shaded region.
Carrier
Lifter
Bridge Walker
Gen 1
Gen 10
Gen 20 1:10.
Gen 1 R:0.0
Gen 15 R:0.1
Gen 30 1.3.3
Gen 1 R:3.8
Gen 5
Gen 15 R.6.6
Figure 4: Evolution of robot designs. For each of the three selected tasks, we visualize the population three different generations. Each column corresponds to one generation which we show four top performing robots along with their average reward.
5.2 Evolution analysis
In Figure we visualize the top four robots in three different generations training the genetic algorithm for the Carrier, Lifter, and Bridge Walker task. We also show the average reward these designs achieve.
In the carrier task, the robot must catch an object that falls from above and then carry that object as far as possible. Therefore, successful design for this task achieves two main goals 1) allowing the

---

Carrier (Easy)
Climber (Medium)
Hand-Designed
Hand-Designed
Optimized Design
Hand- Hand-Designed Designed
Hand- Hand-Deskee Designed
Hand- Hand-Designed Designed
Optimized Design
Slide Beam (Hard)
Hand-Designed
Designd
Hand-Designed
Optimized Design
Figure 5: Comparison between algorithm-optinie robots and hand designed robots on three tasks. In each task, visualize one robot optimized by the algorithm and several hand-designed robots.
robot to catch and hold the object securely 2) allowing the robot to move fast. We observe that robots with a block-holding mechanism and with legs are selected for in the top survivors of generation I (randomly initialized). As evolution progresses, these structures become increasingly optimized. Specifically, in later generations, the robots' structures allow them walk faster while still preventing the block from falling.
similar comparison pattern can be seen in the Lifter task, where the algorithm learns a parallel gripper-like shape underneath the robot in order to manipulate an object. Unlike in the carrier task, the design structures that the algorithm generates are not prominently found the initial generation. Finally, these patterns are echoed in the Bridge Walker task. Here the robot learns to evolve large front foot maximize its surface area and friction force best walk across the soft rope bridge.
5.3 Comparison against hand-designed robots
We compare the performances of robots optimized by algorithm and the hand designed robots several tasks to show the necessity of a co-design algorithm (Figure 5). The structure of the hand designed robots bio-inspired and manually constructed according control are optimized by PPO.
best intuition, and their
For every task, the hand designed robots outperformed by at least one algorithm (usually more). For instance, for the Climber task we tested numerous natural robot designs. However, none of them successfully climbed very far. The issue with designs that could not find the right trade off between getting traction on the wall, and accelerating upwards. The genetic algorithm, however, is able to find this balance. It develops leg-like structures that help the robot make forward progress, as well as long flat back that maximizes contact/frictional forces with the wall. Additionally. the genetic algorithm selects for having hole in the center of its body, which helps it achieve certain optimized walking motion.
For other tasks, the performance between the hand designed robots and the robots produced by the algorithms is quite comparable. This the case with the Carrier robots, as a very natural hand-designed Carrier robot performs almost well the best optimized robots produced by the design-optimizatioo algorithms.
In the final case, there are tasks where neither a hand designed nor robot produced by the algorithm could achieve satisfying performance. One such environment is the Beam Slider environment. For this task, many of the hand design robots fail to even achieve the first part of the goal and position themselves underneath the beam. While there one robot produced by the genetic algorithm that does slide the beam across several pegs, from visual observation we believe it comes nowhere close

---

to exhibiting the optimal behavior in this environment. This suggests that further work is needed in designing co-optimization algorithms that can complete these hard tasks.
Conclusion and future work
In this paper we proposed Evolution Gym, the first large-scale benchmark for evolving the structure and control of soft robots. Through the wide spectrum of tasks in Evolution Gym, systematically studied the performance of current state-of-the-ar co-design algorithms. As a result, we observed how intelligent robots could be evolved autonomously from scratch yet still be capable of accomplishing some surprisingly complex tasks. We also discovered the limitations of existing techniques for evolving more intelligent embodied systems.
There are several potential directions to explored in the future. First, with the help of our proposed benchmark, it is desirable to develop more advanced co-design algorithms to solve the difficult tasks which existing methods cannot address. Our currently implemented baseline algorithms share a bi-level optimization routine where the design optimization is in the outer loop while the control optimization is in the inner loop. However, Evolution Gym is agnostic to the specific training procedure used. As result, some ideas for future work using our framework could include coneur- rently co- co-optimizing the design and control, neuroevolution algorithms, morphogenetic development, gradient-basee methods for design optimization, algorithms with decentralized controllers.
Second, a robot will considered more successful if can perform multiple tasks. Our benchmark suite naturally provides comprehensive set of tasks and can potentially promote more exciting research work about multi-task or multi-objective robot co-design algorithms.
Another consideration is the specific morphological encodings used by the codesign algorithms as more intelligent encodings could lead to better performance. For instance. [38] analyzes the strengths and weaknesses of different morphological encodings. Our baseline algorithms use a direct encoding and CPPN but exploring other encoding representations remains interesting future work.
Finally, since tasks in Evolution Gym are currently limited to either locomotion or manipulation, we plan to further extend Evolution Gym to additional task categories such flying swimming by incorporating new simulation capabilities.
Overall, we believe our carefully benchmarking tool fills an important missing piece in research in soft robotics and robotic evolution algorithms. Armed with the flcxible and expressive framework Evolution Gym provides, we are optimistic that future researchers will use Evolution Gym as a standard test bed to improve co-design methods and evolve more intelligent robots.
Societal Impact
We regard this work as very preliminary piece of research in the field of soft robot co-design, and therefore think that are still far away from causing harm to society. However, we can definitely foresee some problems if this technology were to be applied in the real world on a large scale. For instance, this work may inspire the automatic design of real biological creatures in which serious ethical issues exist. Additionally, since the users have full control over the reward design when customizing the benchmark environments, they could specify pernicious goals and encourage the co-design algorithm to produce more biased results.
Acknowledgments and Disclosure of Funding
We thank Tao Du and the anonymous reviewers for their helpful comments in revising the paper. This work is supported by the Defense Advanced Research Projects Agency SPA8770-200C-0077
References
[1] llge Akkaya, Marcin Andrychowicz, Maciek Chociej. Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube 2019.
with robot hand. arXiv preprint
10

---

[2] OpenAI: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz. Bob Mc- Grew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
[3] Jacob Austin, Rafael Corrales-Fatoo Sofia Wyetzner, and Hod Lipson. Titan: A parallel asynchronous library for multi-agent and soft-body robotics using nvidia cuda. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 7754-7760, 2020.
[4] The GPyOpt authors. GPyOpt: A bayesian optimization framework in python. http:// github. www.coooo////oo http///effielll//PP/OOOO/ 2016.
[5] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
[6] Nicholas Cheney, Jeff Clune, and Hod Lipson. Evolved electrophysiological soft robots. In Artificial Life Conference Proceedings 14, pages 222-229. MIT Press, 2014.
[7] Nick Cheney, Robert MacCurdy, Jeff Clune, and Hod Lipson. Unshackling evolution: Evolv- ing soft robots with multiple materials and powerful generative encoding. SIGEVOlution, 7(1):11-23, August 2014.
[8] Francesco Corucci, Nick Cheney, Francesco Giorgio-Serchi, Josh Bongard, and Cecilia Laschi. Evolving soft locomotion in aquatic and terrestrial environments: effects of material properties and environmental transitions, 2017.
[9] Francesco Corucci, Nick Cheney, Francesco Giorgio-Serchi, Josh Bongard, and Cecilia Laschi. Evolving soft locomotion in aquatic and terrestrial environments: effects of material properties and environmental transitions. Soft robotics, 5(4))475-495, 2018.
[10] Francesco Corucci, Nick Cheney, Hod Lipson, Cecilia Laschi, and Josh Bongard. Evolving swimming soft-bodied creatures. In ALIFE XV, The Fifteenth International Conference on the Synthesis and Simulation of Living Systems, Late Breaking Proceedings, volume 6, 2016.
[11] Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International conference on machine learning, pages 1329- 1338. PMLR, 2016.
[12] Christer Ericson. Real-time collision detection. CRC Press, 2004.
[13] David Ha. Reinforcement learning for improving agent design. Artificial life, 25(4):352-365, 2019.
[14] Donald J Hejna III, Pieter Abbeel, and Lerrel Pinto. Task-agnostic morphology evolution. arXiv preprint arXiv:2102. 13100, 2021.
[15] Jonathan Hiller and Hod Lipson. Dynamic simulation of soft multimaterial 3d-printed objects. Soft robotics, 1(1):88-101, 2014.
[16] Zhiao Huang, Yuanming Hu, Tao Du, Siyuan Zhou, Hao Su, Joshua Tenenbaum, and Chuang Gan. Plasticinelab: A soft-body manipulation benchmark with differentiable physics, 2021.
[17] Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics, 4(26), 2019.
[18] Kirthevasan Kandasamy, Akshay Krishnamurthy, Jeff Schneider, and Barnabás P6czos. Paral- lelised bayesian optimisation via thompson sampling. In International Conference on Artificial Intelligence and Statistics, pages 133-142. PMLR, 2018.
[19] Ilya Kostrikov. Pytorch implementations of reinforcement learning algorithms. https:// github. wwt.u...omikosttiio
2018
[20] Harold J Kushner. A new method of locating the maximum point of an arbitrary multipeak curve in the presence of noise. 1964.

---

[21] Thomas Liao, Grant Wang, Brian Yang. Rene Lee, Kristofer Pister, Sergey Levine, and Roberto Calandra. Data-efficient learning of morphology and controller for a microrobot. In 2019 International Conference on Robolics and Automation (ICRA), pages 148882449) IEEF, 2019.
[22] Alan McIntyre, Matt Kallada, Cesar G. Miguel, and Carolina Feher da Silva. neat-python. https://giz thtb...tthut...g com/ Codeeeecaaimrrr//natt mew.app-ronnaattptoooo...
[23] Eric Medvet, Alberto Bartoli, Andrea De Lorenzo, and Stefano Seriani. 2d- simulation tool for the optimization of 2-d voxel-based soft robots. SoftwareX, 12:100573, 2020.
[24] Zbigniew Michalewiez. Genetic algorithms+ data structures= evolution programs. Springer Science Business Media, 2013.
[25] Močkus. On bayesian methods for seeking the extremum. In G. Marchuk, editor, Optimiza- tion Techniques IFIP Technical Conference Novosibirsk, July 1-7, 1974, pages 400-404 Berlin, Heidelberg, 1975. Springer Berlin Heidelberg.
[26] Andrew Nealen, Matthias Müller, Richard Keiser, Eddy Boxerman, and Mark Carlson. Physi- cally based deformable models in computer graphics. In Computer graphics forum, volume 25, pages 809-836. Wiley Online Library, 2006.
[27] Deepak Pathak, Chris Lu, Trevor Darrell, Phillip Isola, and Alexei A Efros. Learning to control assebbisg morphologies: study of generalization via modularity. arXiv preprint arXiv: 0254666..0..6.. 2019.
[28] Uber Research. neto
[29] Charles Schaff, David Yunis, Ayan Chakrabarti, and Matthew Walter. Jointly learning to construct and control agents using deep reinforcement learning. In 2019 International Conference on Robotics and Automation (ICRA), pages 9798-9805. IEEE, 2019.
[30] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
[31] Karl Sims. Evolving virtual creatures. In Proceedings of the 21st annual conference on Computer graphics and interactive techniques. pages 15-22, 1994.
[32] Andrew Spielberg, Allan Zhao, Yuanming Hu, Tao Du Wojeiech Matusik, and Daniela Rus. Leaming-in-the-logg optimization: End-to-end control and co-design of soft robots through learned deep latent representations.
H. Wallach, H. Larochelle, Beygelzimer, F. d' d'Alché- Buc, F. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[33] Kenneth O Stanley, Compositional pattern producing networks: A novel abstraction of develop- ment. Genetic programming and evolvable machines, 8(2):131-162, 2007.
[34] Kenneth O Stanley and Risto Mikkuulanenn. Evolving neural networks through augmenting topologies. Evolutionary computation, 10(2):99-127, 2002.
[35] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[36] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. Deepmind control suite, 2018.
[37] Merel Van Diepen and Kristina Shea. A spatial grammar method for the computational design synthesis virtual soft locomotion robots. Journal of Mechanical Design, 141(10), 2019.
[38] Frank Veenstra and Kyme Glette. How different encodings affect performance and diversification when evolving the morphology and control of 2d virtual creatures. In Artificial Life Conference Proceedings, pages 592-601. MIT Press, 2020.
[39] Tingwu Wang, Yuhao Zhou, Sanja Fidler, and Jimmy Ba. Neural graph evolution: Towards efficient automatic robot design, 2019.
12

---

[40] Fei Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real- world perception for embodied agents. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9068-9079, 2018.
[41] Allan Zhao, Jie Xu, Mina Konaković-Lukovic, Josephine Hughes, Andrew Spielberg. Daniela Rus, and Wojciech Matusik. Robogrammar: Graph grammar for terrain cerain-optimized robot design. ACM Trans. Graph. 39(6), November 2020.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] The URL is presented in the abstract.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix D.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] We ran experiments with multiple random seeds and reported error bars. See Section
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section 5.
4. If you using existing assets (e.g., code, data, models) or curating/releasing
assets... (a) If your work uses existing assets, did you cite the creators? [Yes]] The existing code
implementation for our bascline algorithms are cited in Section 4.
(b) Did you mention the license of the assets? [Yes] This benchmark platform will be released under the MIT license. See Section 1.
(c) Did you include any assets either in the supplemental material URL? [Yes] The URL presented in the abstract.
(d) Did you discuss whether and how consent was obtained from people whose data you'le 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "X-0XOPY_a4", "reviewer": "Reviewer_j4pt", "summary": "The paper proposes a framework for benchmarking algorithms for optimising both bodies and controllers of soft-bodied robots. It also presents three optimisation solutions and evaluate them on a suit of benchmark tasks. The authors also evaluate some hand-coded solutions; the performance of these is worse for some tasks, while about the same for others.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main novelty concerns the benchmarking framework, called Evolution Gym. The challenge here is to define representations that are sufficiently generic to define a range of environments and tasks, robot morphologies and controllers. This is addressed by extending existing approaching in meaningful ways.\n\nThe paper is overall well presented.\n\nMajor comments:\n\nThe Evolution Gym has merit in offering possibilities to benchmarking solutions. The robot morphologies and designs are fairly intriguing. Yet, I am not convinced Evolution Gym is a suitable tool for solving real-world problems with soft robotics. Evolution Gym is limited to 2-D. Even Sim's work from 1994 (the tasks of which are described as \"simple\" by the authors) was conducted in 3-D.  Another simplification is the need for discretisation (e.g. you cannot have a tilted plane as the ground). Moreover, the treatment of sensing seems rather simplistic. While \"observations\" are considered, it is not clear how these would be obtained in practice (e.g. position of corner modules relative to the barycentre of the soft-bodied robot). There does not appear to be noise on actuation or sensing. The only source of uncertainty - to my understanding - is the initial placement of the object in the manipulation tasks.\n\nReal-world scenarios are also likely requiring a robot to deal with a range of object geometries to be manipulated. To my understanding the objects that are manipulated in the same task are of constant shape and size. On a similar note, it would have been interesting to define the framework more general, so that the same robot morphologies can be tested not on a single but a set of tasks.\n\nRegarding Algorithm 1, why it is necessary to start the controller evolution from scratch every generation (as $S$ is only passed as an argument to OptimizeDesign, but not to OptimizeController). I understand that not every controller will be compatible with every morphology. However, there could be ways to deal with this.\n\nThis raises another question, why having an outer and inner loop at all for morphology and design optimisation? Why not co-evolve them as part of the same genotype? Maybe your approach offers better \"evolvability\"? In any case, it should be made clear that it is up to the user to define their optimisation solutions.\n\nFor the benchmarking framework to be widely adopted, it would be good to understand how it can be extended. Moreover, it would be good to have some tasks with scalable complexity, thereby allowing the community to progress along the same \"axis\", rather than each having to invent their own tasks.\n\nThe actions need to be explained more precisely. I guess that each component of the action vector is associated with an actuator voxel (not each \"value\" of the action vector as is stated).\n\nI would have appreciated more detail on the controller design, and how it combines sensing with actuation (I only gave the supplementary material a quick check, \"actions\" are no mentioned therein). For example, there is a sensing window of size 2W providing terrain information; but up to which height, for example, for the Slider task in Figure 2, where the robot needs to perceive objects above it? Related to this, I wonder, which of the tasks, if any, require closed-loop control, and how is it realised? This I was missing from the descriptions in Section 3.3.\n\nYou say twice \"general and universal representation for various categories\", which is somewhat contradicting. How universal is the representation? You state that the shape of a robot can change as voxels can flip from/to empty; can the bounding box of the robot evolve as well? Can the building blocks be combined in arbitrary ways (e.g. horizontal actuator on top of vertical actuator) provided the structure remains connected? Why can objects not have empty cells (holes) - wouldn't this impact on inertial forces? \n\nIt seems that each optimisation is run only 3 times per task (Figure 3 states \"3 different random seeds\"). This would not be sufficient for a rigorous comparison.\n\nI like the finding that a simple GA was possibly the best performing solution (though could be due to the limited sample size). This would show how important benchmarking is. However, there is the risk all the effort is spend on getting the perfect optimisation algorithm, whereas the real problems would relate more to how the robot can best sense its environment and move therein - and some of this is pre-determined/rigid given the representations that has been chosen.\n\nThe paper consistently talks about \"optimal\" control/designs. There is nothing to suggest that any of the solutions are optimal.\n\nMinor comments:\n\nIt is stated that the \"robots are evolved autonomously from scratch without prior knowledge\", however, designing the reward function \"should be defined to precisely characterize the robot's completeness of the task\" at every step.\n\nI would place Figure 1 on top to save place.\n\nTypo - \"Speilberg\"\n\n\"combining the state-of-the-art design optimization strategies..\" - I would remove \"the\"\n\n\"with of steps of\"\n\nI found the description of the algorithm first confusing, as the line with OptimizeControl is described as \"Optimise the controller\" whereas the one with OptimizeDesign is described ass \"Propose [..] a robot design\".\n\nIn Figure 3, the y axis ranges are not consistent, making it hard to compare the solutions.\n\nUPDATED (following rebuttal):\n\nI thank the reviewers for their responses. I am still concerned by the limited number of independent trials (3, or possibly 6 in the camera-ready version). This would seem to set low standards, especially for a benchmark paper. Moreover, it is not clear how increasingly complex problems could be designed. The authors state that \"ranking [their] tasks by difficulty level achieves a similar purpose\". However, with only a limited number of tasks, the complexity would be bounded, and the ranking is somewhat subjective.\n\nI have changed the overall rating from 5 to 4.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think this is not applicable for this work.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "LQS9ipB2Umm", "reviewer": "Reviewer_523c", "summary": "The paper proposes a benchmark for Evolving Soft Robots. The benchmark is designed to facilitate research in co-design of control algorithms and robot morphologies. The proposed framework relies on a custom physics engine which allows a simple mass-spring-damper connection of individual modules. The proposed benchmark is suitable for both manipulation and locomotion even though restricted to 2D environments only. Authors provide also an initial evaluation of SoTA design optimization algorithms (genetic algorithm, bayesian optmisation, CPPN-NEAT) on the proposed benchmark. ", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The submitted paper is well written, easy to read and very clear in the writing. To my knowledge, this is the first attempt to provide a benchmark for evaluating co-design on soft robots . Moreover the proposed approach encompasses both locomotion and manipulation. The major limitation of the submitted paper is the significance to the research community. The proposed environment is extremely artificial. First of all it's just a 2D-only environment and a 3D environment would be more appealing for potential adopters of the benchmark. Moreover, the environment doesn't seem suitable to model rotational joints and rotational electric motors which are at the moment the main components for industrial robots. \n\n= MAJOR COMMENTS =\n\n- The proposed environment seem to have important limitations which might hinder adoption from users and therefore the impact on the community. Authors should try to address these limitations or at least conduct experiments to provide evidence that these are not limitations. First limitation is the environment being only 2D; this is a major limitation since applications require 3D modeling. The second limitation is the lack of rotational component;  most of the proposed robot-designs and robot-embodiments are a mixture of soft and rigid components. Third limitation is that authors do not provide evidence that their benchmark can cope with rigid only embodiments; even though soft robotics is an emerging field of research, most of existing industrial applications still rely on assuming rigid only components and testing the physics engine on rigid only robot-designs might attract additional interest. Last but not least, authors have tested the benchmark with a limited number of voxels in the order of 100 voxels; however interesting applications might require orders of magnitude more voxels (1000 to 10000) and no analysis is provided on how the proposed physics engine and computations will scale with the number of voxels.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I don't see negative potential societal impact for the proposed benchmark. ", "review_text": "", "extra_scores": {"ethical_concerns": "As far as I can tell, no ethical concerns apply on this submission.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5h"}, "extra_text": {}}, {"review_id": "FDceoMYIUWo", "reviewer": "Reviewer_nQ7Q", "summary": "The paper presents Evolution Gym, a benchmark for the evaluation of design and control co-optimization algorithms on simulated voxel-based soft robots. It contains a back-end soft-body simulator (written in C++ with bindings in Python), a set of 30 evaluation tasks of various types and difficulty levels, 3 design optimization algorithms and 1 control optimization algorithm. The algorithms are evaluated on the benchmark tasks and the results are reported in the paper.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is very well written and structured. It comes with a well-furnished supplementary including an appendix detailing all the benchmark tasks, a video showing the robot designs and control policies found by the co-optimization algorithms, the source code, as well as a website where the user can visualize all the benchmark tasks. \n\nThe diversity of the proposed tasks is impressive and covers a wide range of challenges for design and control co-optimization algorithms. The paper compares the performance of three different design optimization algorithms coupled with a single control optimization algorithm on 10 selected tasks (in the main text, more evaluation in the appendix).\n\nFor these reasons, I think the paper is a valuable contribution to the Neurips community. It is important to note, however, that the paper mostly focuses on morphological optimization, control optimization being limited to a single DeepRL algorithm (PPO). As its name indicates, the paper would therefore be more relevant to the Evolutionary Computation community (e.g. in conferences such as GECCO). Nevertheless, it highlights a number of original and interesting challenges that are definitely relevant to Neurips. \n\nMy main concerns about the paper are the following (see below for detailled comments on each point):\n - The proposed benchmark imposes a number of constraints that could limit its generality.\n - The discussion on the limitations, perspectives and societal impact of the paper is very short and does not address important points.\n\nI also suggest to the authors to position their contributions wrt to these related papers:\n- https://pathak22.github.io/modular-assemblies/resources/assemblies.pdf\n- https://arxiv.org/abs/1906.05370\n- https://arxiv.org/abs/1810.03779\n- https://direct.mit.edu/isal/proceedings/isal2020/32/592/98402\n\n## The proposed benchmark imposes a number of constraints that could limit its generality\n\nDespite its name (Evolution Gym), the proposed benchmark is much more constrained than OpenAI Gym. The main contribution of OpenAI Gym (imo at least) was the proposition of a standardized software interface allowing to easily plug any RL algorithm to any sequential environment. It was completely agnostic to e.g. the definition of observation and action spaces, or to the state space and transition functions of the environments.\n\nEvolution Gym instead imposes a number of hard constraints to the user (at least in its current version):- The environment dynamics is fixed and based on 2D voxel-based soft robots.\n- The specification of the robot structure is fixed (Section 3.2).\n- Observation and action spaces are mostly pre-specified (section 3.3).\n- The co-optimization training procedure is fixed: First the design optimizer proposed a new robot structure to the control optimizer, then the control optimizer computes the optimal controller for the given structure (Section 4).\n\nThese constraints could be discussed in more detail in the paper, in particular in the Conclusion:\n - Can the benchmark be easily extended to other types of environments? (i.e. not only voxel-based soft robots, see e.g. the papers mentioned above for possible alternative environments).\n - How can the user define a new task? What amount of code does it involve? What is the software interface? It would be interesting to provide a tutorial in the supplementary material or on the website.\n - Can we evolve/train a robot on a distribution of tasks? (in a meta-learning fashion)\n - Can it be applied to morphogenetic development? (i.e. where the robot structure grows during the lifetime of the robot)\n - Is it possible to easily define other observation and action spaces? What are the limitations in this respect? For instance, is it possible to optimize a decentralized controller? (where each voxel optimizes its own control policy based only on local observation). Or is it possible to specify new sensors, e.g. a light sensor?\n- Can the design and the control be jointly optimized? (instead of the currently implemented 2-phase optimization procedure: first design, then control). \n\nIt would have been interesting to evaluate more than one control optimization algorithm (currently: PPO). This would have allowed to evaluate interaction effects between design and control optimization (i.e. which control optimization algorithm is better suited for a given design optimization algorithm). Neuroevolution approaches allowing to jointly optimize network architecture and weights (e.g. NEAT or variations of it) would be of particular interest here, as it would challenge the currently imposed 2-phase optimization procedure.  \n\n## The discussion on the limitations, perspectives and societal impact of the paper is very short and does not address important points.\n\n- As mentioned above, there are many limitations and perspectives that would deserve to be discussed in the paper.\n- There is no mention of societal impact, whereas there are important points to be mentioned here as well (related e.g. on the automatic design of biological creatures)\n\n## Minor comments\n- Fig 2: it could help the reader to precise in the caption that the verbal description of the tasks is provided in section 3.5.\n- The taxonomy of tasks is not super clear (Locomotion vs. Object Manipulation) for several reasons. First, as mentioned line 191, some tasks are a mix of both. Second, there are other types of tasks in the appendix that are not mentioned in the paper (e.g. shape change). Maybe using labels instead of categories would help (i.e. a task can be associated with multiple labels).\n- Lines 327 to 329 suggest that an optimal reward exists for each task. If it is the case, it would be interesting to plot the optimal reward on Fig 3 (and in similar figs in the appendix) in order to better evaluate how well the optimization is able to approach that optimal.\n- Lines 298 to 302. This is hard to see this from the Fig, where all but one designs have the mentioned two structures (i.e.  two legs and a structure up top that prevents the block from sliding off).\n- Section 5.3: You could say a bit more about how you performed hand design, or at least discuss this point in the discussion or appendix. What procedure did you follow? What are the limitations? What would be a more thorough procedure? (e.g. asking N participants to perform the hand design by imposing some constraints on how to do it, e.g. limited time).\n- Regarding the video provided in the supplementary material:\n    - I was surprised that most evolved designs are non-symmetrical, even in tasks that are inherently symmetrical like BiderectionalWalker  or Climber. This seems to contradict the claim made in the abstract that robots \"often grow to resemble existing natural creatures\". It could be interesting to further discuss this point.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See the section \"The discussion on the limitations, perspectives and societal impact of the paper is very short and does not address important points\" in the main review.", "review_text": "", "extra_scores": {"ethical_concerns": "No major concerns. See however the section \"The discussion on the limitations, perspectives and societal impact of the paper is very short and does not address important points\" in the main review.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "4tKBxT4yeE9", "reviewer": "Reviewer_VGKK", "summary": "This paper introduces a simulator and suite of benchmarks for studying the co-design of soft-robots and their controllers. It provides a simplified abstraction to describe soft robots from their components, a corresponding physics engine, and a set of tasks and associated metrics to optimize.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThis is a solid paper. It is well organized, describes the problem it aims to tackle clearly, and sufficient details about the various assumptions and simplifications that were made, the benchmark tasks, and a solid set of baselines to work from.\n\nA few things I'd wished to see:\n\n- more details about the system and computational requirements. As a user, I'd love to get a sense of what I'd be getting into if I were to attempt tackling such benchmarks.\n\n- the code! The paper says the APIs are OpenAI-gym-like, and I would have loved to be in a position to evaluate the usability of the benchmark suite first-hand.\n\n- Perception being considered as part of the design space. The agents have access to the full state of the robot and to a local window of the environment. This may be a case of 'too little and too much at the same time': many of the tasks mean the controller needs to reason about contact, which it only has access to indirectly through geometric reasoning from the state, which is hard. Enabling the robots to have access to summary state like contact information, or range sensors, could actually lead to improved designs, faster learning, and would also make the benchmark one step closer to being useful for more realistic embodiments which will never have access to the full state of the system, particularly if they're soft.\n\n- Any suggestion that insights that are transferrable beyond this specific sets of benchmarks can be derived from this co-design problem setting. Can we learn controllers that facilitate the selection of candidate physical designs faster? Can we quickly validate candidate designs via simple controllers and iterate faster? Lots of possible questions to ask, and which would enhance the value of the overall benchmark suite.\n\nNone of those issues however detract from the fact that this is an interesting benchmark. I will flag my review as comparatively lower confidence because I don't know the landscape of the existing tooling around morphology design and soft robots, so I can not properly judge how unique this contribution is.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See review.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Meta-review for "Evolution Gym: A Large-Scale Benchmark for Evolving Soft Robots"

While policy optimization is a well-studied topic in both control and RL, little work has been done on robot-design optimization, in particular, joint optimization of the robot agent and its policy. This work is motivated at filling the gap, and proposes a benchmark for evolving (both policy and controller) of soft-bodied robots (a rich area in evolutionary computation, but rather less explored in the RL community). Rather than rely on detailed and computationally expensive simulation, they opted for simplicity and created a benchmark using a simple, but fast 2D physics engine (a mass-spring system in C++) to implement a suite of soft-bodied robot environments with OpenAI-Gym like interface. They provide sufficient details on the assumptions and simplifications made in the simulation. See Supplementary Materials for examples: https://sites.google.com/view/evolution-gym-benchmark/all-tasks/object-manipulation

Reviews were mixed, with two reviewers giving 4’s and 5’s, and two reviewers giving 7 and 8’s. That being said, all reviews agreed that the paper is well written, and is a novel attempt at providing a benchmark for evaluating soft-robots.

The main criticism of the work raised is that the current environment is not realistic enough to be useful for real robot design, hence the significance to certain parts of the research community is reduced. The counter argument raised by positive reviewers (and authors) is that the simplification is actually intentional, and that everyone including authors agree that the proposed benchmark is very much constrained (limited to 2D voxel-based soft robots, with specific observation and action spaces). As one reviewer put it: *The authors agree that there are strong limitations and argue that this doesn't limit the interest of the benchmark because many of the proposed tasks are still not solved by the benchmarked algorithms.*

To improve the work, reviewers have proposed that the authors address the simplicity by not necessarily making it more realistic and useful for realistic robot design, but rather keep the simplicity, but make the framework really extensible, so it can easily be extended as needed by users. For instance as raised by nQ7Q, *One way to do it would be to provide a set of tutorials (e.g. as Jupyter Notebooks) explaining how to proceed for each of the possible extensions mentioned above. Show that this interface is general enough to easily extend the benchmark in the directions pointed out by the reviewers. This can be done by providing use-case examples on how to replace the simulator by another one (e.g. Mujoco?) ; How to design a new task? ; how to implement new observation and action spaces? ; how to change the training procedure?* The authors have acknowledged the suggestion and opted to provide comprehensive tutorials and detailed API documentation.

At the end of the day, discussions from both sides didn't converge, and while reviewers agreed to disagree, they recognized each other's points. While the current benchmark is likely not going to be of any direct use for designing real world physical robots, the proposed benchmark is a solid attempt at providing the first co-design benchmark, a break from the traditional pure policy optimization RL benchmarks, and despite its simplicity, the tasks proposed still cannot be solved by current methods. Arguably, like the original OpenAI gym, the simplicity, and speed of execution might outweigh direct real world applications, and instead help grow the co-design research community.

In this AC's view, there are already numerous Gym environments that also emphasize on photorealism and high quality realistic physics simulations (including accurate fluid and mesh simulations), but those environments haven't really gained traction from the community possibly due to the high barrier to entry in terms of compute, slowing down research iteration, leading us to be where we are in RL. The RL community (perhaps not the robotics community) will have something to gain from this co-design benchmark, which may help facilitate a move towards solving problems that is not pure policy-optimization, and perhaps this is what's needed to move RL research into new directions to help break away from the local optimum we are hill-climbing together as a community. Furthermore, the benchmark may act as a bridge for the NeurIPS RL community and the evolution / co-design folks and help facilitate a common language / platform to exchange ideas, results and discussion. In my view, the speed and simplicity will also help improve accessibility to folks without access to SOTA compute facilities, which may help bring in additional talent in (as we have seen with OpenAI Gym).

For these reasons, I'm recommending acceptance of this work, despite the borderline score of 6, because I do believe the benchmark will be beneficial to the NeurIPS community at large.

Decision: Accept (Poster)