Evolution Gym: A Large-Scale Benchmark for Evolving Soft Robots

Abstract: Both the design and control of a robot play equally important roles in its task performance. However, while optimal control is well studied in the machine learning and robotics community, less attention is placed on finding the optimal robot design. This is mainly because co-optimizing design and control in robotics is characterized as a challenging problem, and more importantly, a comprehensive evaluation benchmark for co-optimization does not exist. In this paper, we propose Evolution Gym, the first large-scale benchmark for co-optimizing the design and control of soft robots. In our benchmark, each robot is composed of different types of voxels (e.g., soft, rigid, actuators), resulting in a modular and expressive robot design space. Our benchmark environments span a wide range of tasks, including locomotion on various types of terrains and manipulation. Furthermore, we develop several robot co-evolution algorithms by combining state-of-the-art design optimization methods and deep reinforcement learning techniques. Evaluating the algorithms on our benchmark platform, we observe robots exhibiting increasingly complex behaviors as evolution progresses, with the best evolved designs solving many of our proposed tasks. Additionally, even though robot designs are evolved autonomously from scratch without prior knowledge, they often grow to resemble existing natural creatures while outperforming hand-designed robots. Nevertheless, all tested algorithms fail to find robots that succeed in our hardest environments. This suggests that more advanced algorithms are required to explore the high-dimensional design space and evolve increasingly intelligent robots -- an area of research in which we hope Evolution Gym will accelerate progress. Our website with code, environments, documentation, and tutorials is available at http://evogym.csail.mit.edu/.


Reviews and author responses:

 Thanks for addressing the stated concerns in detail in your response. Specifically, thank you for point (1). Though it's still not completely clear to me that voxel-based robots are going to be especially easy to fabricate in the future, the larger tie-in to modular structures is clear and justifies voxels enough.

Further, thank you for point (7); it clarifies that this simple, less-physical and faster simulation approach is chosen to in a sense "match" the capabilities of many emerging techniques (which can already have difficulties in 2D space). I would very strongly suggest adding some of your language from point 7 to the manuscript if possible.

As a result of this response I will increase my score from 5 to 6.




This paper proposes Evolution Gym, a simulation environment and benchmark which may be used for evaluating design and control optimization techniques for soft robots built in a voxelized design space. In addition to the benchmark, the paper compares several existing design and control optimization techniques for a common set of tasks.  The paper posits that co-design algorithms are understudied due to the complex, bi-level nature of the underlying problem and the lack of a benchmarking platform. It seems that a benchmarking platform which could be used for co-design of robots would be a useful tool to roboticists. In my opinion, such a platform could be used to gain understanding of some key concepts underlying soft robots (e.g. the role of morphological intelligence) while providing clear baselines of optimization techniques and behavior which will establish a state-of-the-art. This contribution has the potential to be important to soft robotics.

However, some aspects of the provided benchmark and comparison are underdeveloped. In general, a stronger case could have been made for the particular choice of physics, design space, and the presence of feedback in the proposed benchmark. Further, some clarifications are needed about the optimization comparison.

1. The approach given has a key sim-to-real limitation that is not discussed. Details are given in the following section.

2. Why are voxel-based design spaces the best choice? It is claimed that voxel-based soft robots offer a "general and universal representation" of various "categories" of robot designs, yet it is not clear which categories the learned robots fit into. Indeed, it is not even clear at present how to physically construct voxel-based soft robots under this general representation (e.g. Kriegman et al, 2020, Scalable sim-to-real transfer of soft robot designs). Authors might consider commenting on learned systems, noting whether actuators presently exist that allow these robots to be constructed. Is it better to have a generalized representation such as this one, or many possible constrained design spaces that are more directly inspired from existing soft robot designs? Could robots learned in Evolution Gym guide soft actuator development? Detailed discussion of this point could help readers.

3. Why does a benchmark need to include feedback control? The paper notes that prior co-design work in soft robotics mainly has mainly explored open-loop control, while the proposed framework allows detailed high-dimensional observations to be used as feedback. This engages directly with ideas of morphological computation (e.g. Tolley and Rus, 2015) and its role in soft robotics; the simulator could allow experiments about which balance between morphological computation and feedback is best. Yet, here, it is not clear how much feedback the learned controllers are using and whether the inclusion of it was needed. Comments about this issue, and how the proposed benchmark may help with its study, would better show the impact of the work.

4. It seems somewhat surprising that the plain GA outperforms the other methods in several tasks, given that GAs have not been shown to converge to global minima. Comments on this success might help readers.

5. Is it possible to use gradient-based methods for design optimization with this benchmark? Would anyone need to do so in the given design space? Please clarify.

6. Why is a simplified, 2D spring-based physics representation best? It is clear that such an approach is faster computationally than other mechanics methods (e.g. Finite Element Analysis or particle-based methods), but accuracy also suffers if the approach is simplified. Why not take a more computationally heavy (but accurate) approach and wait for Moore's Law? Please discuss. Further, does this physical simulation differ from the one used by Medvet et al 2020 (citation 21)? Please clarify. One key limitation of a this simplified benchmark environment is that its gap with reality is not yet characterized. The paper states that "we observed how intelligent robots could be evolved autonomously from scratch yet still be capable of accomplishing some surprisingly complex tasks." While this did happen in the simulation environment, it is not clear that real devices built with the best-performing designs and controllers would truly perform best in the real world. As noted above, it is unclear whether the result robots are easy to construct, and it is further unclear whether sensing technologies are possible for the type of observations described in Section 3.3. While a detailed experimental validation of all proposed robots is clearly out of scope, some comments on the reality gap and the transition from 2D to 3D are 
necessary given the embodied nature of the problem.

 Dear reviewer 1ku4,

Thank you again for your feedback!

First, we agree that the strengths of the benchmark can be illustrated in more detail, however, we want to emphasize that the evaluation is still important and is a standard practice in benchmark papers [1, 2]. In our work, the evaluation serves two key functions. 1) Given the current state of co-design algorithms, the evaluation validates that our tasks are feasible, comprehensive, and of reasonable difficulty. 2) The evaluation demonstrates that state-of-the-art algorithms fail on some tasks, which necessitates such a benchmark to motivate more work in this field. 

In a modified version of the paper, we promise to make the following changes to demonstrate the strengths of our benchmark better:

1. Provide detailed documentation of our open-source, freely available code so researchers can easily begin experimenting and update our website with additional descriptions and images.

2. Modify the text of the main paper to include more thorough explanations and justifications of the strengths of our pipeline (according to the explanations we included in our first response). Specifically, we would elaborate on our pipeline’s user-friendliness, the advantages of the morphological encoding used in this benchmark, the advantages of using this framework in comparison to existing ones, and a justification of our choice of the physics-based simulation engine.

We hope this addresses your concerns, and please let us know if there is anything else you believe would be important for us to modify and/or include in the camera-ready version. We would be happy to answer any remaining questions you may have.

Sincerely,

The authors of Evolution Gym

[1] Lin, Xingyu, et al. “SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulation.” Conference on Robot Learning. 2020.

[2] Cobbe, Karl, et al. “Leveraging procedural generation to benchmark reinforcement learning.” International conference on machine learning. PMLR, 2020.


 Thank you for your response. 

I agree, the proposed simulator and benchmark are a very valuable contribution to the community. However, I still believe that the paper does not reflect this at the moment by focusing on an experimental evaluation that provides limited new insight, instead of focusing on the strengths of the benchmark. 

Based on this, I will maintain my current rating. 

 Dear reviewer, 

Thank you for your feedback and constructive suggestions. We are pleased to hear you found our paper well-written and that you see the novelty of our methods. Below, we have addressed the main concerns in your review.

**User-friendliness of the interfaces, and comparison with other tools**

First, it is a bit difficult to compare the user-friendliness of our tool since there are not many other tools to compare it with. To our knowledge, our framework is the first of its kind to provide a benchmark in soft robotics for co-design algorithms. Nonetheless, we built our framework off of the tools provided by OpenAI Gym, which means it inherits OpenAI Gym’s structure and functionality and is easy to get started with for its potential users. Both the control and design parts of our framework allow the user to plug and play different algorithms. To ensure users are able to easily use our tool, we will provide detailed documentation about all the APIs and show small simple examples to explain how to use the benchmark system upon publication of the paper.

**What are the limits, or advantages on the expressivity of the morphological encoding used in this benchmark compared to the literature?**

We chose a discretized, voxel-based design space because it provides us with a balance of expressiveness and efficiency. As the number of voxels grows linearly, the size of the design space grows exponentially. As a result, with only several different voxel types and < 100 voxels for each robot, we are able to construct a wide diversity of robot morphologies. Further, the morphological representation depends on the simulation methods. The voxel-based representation in our work can be simulated by the fast mass-spring system simulation while other soft body representations such as meshes require more computationally expensive FEM simulation. Such a computationally cheap simulation allows our framework to be efficient enough to train robots in a matter of minutes and provides a computationally tractable benchmark for fast evaluating and iterating co-design algorithms. Finally, this choice of representation is a common practice for soft robots as evidenced by several other related works in the field (Medvet et al. 2020 [1]).

**What are the advantages of using this framework compared to other existing ones?**

First, to our knowledge, our paper is the first to provide a comprehensive benchmark for evaluating co-design in soft robotics. Consequently, our paper fills an important niche in the soft robotics community that existing frameworks have not been able to provide. 

Second, unlike existing work, our paper introduces a diverse range of tasks categorized into several difficulty levels providing a comprehensive evaluation of the algorithms and also a challenge to algorithm development. We are optimistic that future researchers will be able to rapidly improve co-design algorithms armed with the flexible and expressive framework EvoGym provides.

**Experimental Comparison**

Thanks for pointing out this issue.  We discuss the experiments for three reasons:
1) To analyze the designs found by baseline algorithms (since analysis of the designs is also a goal of our benchmark)
2) To compare existing state-of-the-art baselines (another goal of our work)
3) To show the insufficiency of the current methods and demonstrate the necessity of our work

In the revised version, we would be happy to reduce the length of the experimental comparison section to make the focus of this work more clear on the benchmark side.

---------------------------------------------------------------------------------------------------------------

Overall, our carefully designed benchmarking tool fills an important missing piece in research in soft robotics and robotic evolution algorithms. We believe NeurIPS would provide the perfect launching pad to expose our tool to the types of researchers equipped and excited to utilize such a benchmark.

We hope in this response we have clarified the goals of our original paper and adequately addressed your concerns. Your comments have helped us strengthen the work. If you have any other questions, we would be open to further discussion. 

Sincerely,

The authors of Evolution Gym

[1] Medvet, Eric et al. “Biodiversity in Evolved Voxel-based Soft Robots.” GECCO (2020).

[2] Du, Tao, et al. "DiffPD: Differentiable Projective Dynamics with Contact." arXiv preprint arXiv:2101.05917 (2021)


 Dear reviewer, 

Thank you for your feedback and constructive suggestions. We are pleased to hear that you see the novelty of our work. Below, we have addressed your comments in detail.

**(1) Inaccuracy on lines 30-31.**

We assume Hejna et al. 2021 refer to this paper: "Task-Agnostic Morphology Evolution". 

Thank you for pointing out this issue. The message we wanted to convey here is that in the soft body domain (which is the main focus of our paper), co-design has only been attempted on some very simple locomotion tasks so far. Hejna et al. focus on the rigid robot where the degree of freedom of the design space is much smaller than the soft robot in our work. We will update the expression in the paper to make it more accurate.

**(2) Time analysis.**

Simulation speed is very dependent on the type of simulation. For instance, our simulation speed is slower than most rigid-body simulations, like mujoco. This is because there are many more degrees of freedom in soft body simulation compared to a standard rigid body simulation.
	
Another insightful comparison might be with soft, 3D FEM-based simulations. For example, in Du et al. 2021 [1], one simulation step consumes on the order of several seconds to minutes and the PPO training for a single robot design takes on the order of hours to converge. By comparison, in our framework, a single robot can be simulated with hundreds of steps per second and trained on a 4-core machine in a matter of minutes. This highlights the advantage of our 2D mass-spring approach.

Finally, the best comparison of our work would be other 2D voxel-based simulations. For example, our speed is comparable to Medvet et al. 2020 [2]. Both of us represent the environment as a mass-spring system and adopt a standard implementation of the simulation. The main difference is that our simulation is implemented in C++ which is faster than their version in Java. For quick reference, our performance on 25-100 robot voxels + 100 environment voxels using a single core of a standard i7 laptop ranges from 250-150 simulation steps per second. This speed can be further increased with parallelization. 

**(3) 2d vs 3d.**

Extending our framework to handle 3D environments is feasible, however, we believe the 2D nature of our environment is one of its strengths. 2D soft-body simulation provides enough complexity of the task while being much more computationally cheap compared to 3D soft-body counterparts (as demonstrated by our comparison with Du et al. 2021). With such speeds, we believe that future research endeavors in co-design and control optimization algorithms would be more computationally tractable. 

Further, as evidenced by our experimental results, the state-of-the-art co-optimization algorithms fail to search successful robots on many of our hard tasks, even in a simple 2D environment. In other words, the limitations of the learning algorithms stopped us from increasing the sophistication of our environment. However, we believe that 3D extension is an important future work when the learning algorithms become more intelligent and sample efficient.

----------------------------------------------------------------------------------------------------------

Overall, our carefully designed benchmarking tool fills an important missing piece in research in soft robotics and robotic evolution algorithms. We believe NeurIPS would provide the perfect launching pad to expose our tool to the types of researchers equipped and excited to utilize such a benchmark.

We hope in this response we have clarified the goals of our original paper and adequately addressed your concerns. Your comments have helped us strengthen the work. If you have any other questions, we would be open to further discussion. 

Sincerely,

The authors of Evolution Gym

[1] Du, Tao, et al. "DiffPD: Differentiable Projective Dynamics with Contact." arXiv preprint arXiv:2101.05917 (2021)

[2] Medvet, Eric et al. “Biodiversity in Evolved Voxel-based Soft Robots.” GECCO (2020).


 Dear reviewer, 

Thank you for your feedback and constructive suggestions. We are pleased to hear you see the novelty of our methods. Below, we have addressed your comments in detail.

**(1) Justifying a voxel-based design space**

We focus on voxel-based soft robots in this work with inspiration from real cell-based organisms in nature. Such discretized voxel-based design space provides us with the balance of expressiveness and efficiency needed for a benchmark suite. With only several different voxel types and < 100 voxels for each robot, we are able to construct a wide diversity of morphologies; many trained robots are able to perform complex motions and complete difficult tasks. Further, the voxel-based design enables fast simulation which allows our framework to be efficient enough to train these robots in a matter of minutes. 

In addition, voxel-based designs actually provide a fabrication advantage, and modular, discretely-assembled robotics is an active and budding area of research. While you are right to point out our robots cannot be directly fabricated from our current framework output (namely, they are only 2D designs), 3D voxel-based soft robotics is not a far-away reality. Previous work in voxel-based robotics has shown it is possible to fabricate both soft and rigid robots with a wide range of abilities from a simple set of building blocks [1-5]. 

To address your question on soft actuator development -- during our evolution, there were some insightful structures that evolved. For example, we noticed that clear rotational joints emerged automatically through the design process. We believe that these evolved designs can be useful to get an intuitive sense of how to design soft actuators.

**(2) Feedback control**

Our benchmark is agnostic to the type of controller used -- EvoGym is compatible with both feedback and open-loop control. So feedback control is not necessarily required here. However, we mainly use closed-loop control (specifically, deep RL) in the experiments because it is a more general control strategy for both stochastic and deterministic tasks. For example, our closed-loop controller can handle the randomness in Catcher-v0 while the open-loop controller would be infeasible in this case.

That being said, we also believe that interesting future work could investigate how much feedback is necessary for different tasks. This analysis could even be a stepping stone to developing more advanced co-design algorithms, which is exactly the type of research that we hope our work will inspire.

**(3) Surprising success of GA**

This is a great question. As we were also surprised by GA’s performance, we think that might be because of 1) the well-established mechanism of GA and 2) our simple yet effective mutation operator as described in Section 4.1. Additionally, we have found that in comparison to GA, BO does not scale well to high dimensional design space, and CPPN struggles in tasks where complex structures are required since CPPN prefers simple and regular structures. Ultimately, we believe there is a huge space for future improvement in co-design methods, and that the algorithms we present as baselines should in no way be seen as optimal.

**(4) Gradient-based methods for design optimization**

We do think it is possible to use gradient-based methods for design optimization however it might be difficult to obtain the gradient. Currently, the gradient is not available in our implementation because RL is used to evaluate the design which is not a differentiable process. Even though approximating the gradient should be feasible through finite-difference or by fitting a differentiable surrogate model, whether the approximated gradient will speed up the co-design process is a fundamental open question that we hope our work will inspire answers to.

**(5) Justifying a simplified, 2D spring-based physics representation**

Certainly, the 2D spring-based representation is not accurate compared to FEA, however, we believe the speed of the simulation is crucial to our benchmark. This is because our goal is to provide the soft robotics community with a computationally tractable benchmark to quickly evaluate co-design algorithms and encourage algorithm development in this field. But in the future, we are happy to add more sophisticated simulations if necessary, and we plan to continue developing and maintaining them.

**(6) Comparison to Medvet et al. 2020**

The simulation used by Medvet et al. 2020 is similar to ours. Both use a voxel-based representation, simulate objects in 2D, and have comparable speeds. We also both work with relatively standard implementations for mass-spring systems. However, the main contribution of Medvet et al. is its simulation; our main advantages over Medvet et al 2020 are (1) a comprehensive set of well-designed tasks with various difficulty levels to provide the first comparison platform for evaluating co-design algorithms, (2) our implementation of state-of-the-art algorithms to establish a baseline for co-design, and (3) our ability to interface with standard python ML libraries through python bindings for our physics simulation.

**(7) Real-world applicability**

While we provide an analysis on real-world fabricability in comment (1), at the current stage, our framework is not intended to be directly used as a platform for real applications. Rather, its fast simulation speed and simple morphological representation allow it to be an effective testbed for co-design algorithms which can later be used in a more sophisticated and time-consuming system for real applications. Our experiments demonstrate that this framework is needed: current state-of-the-art algorithms often fail to discover feasible 25-50 voxel solutions on some of our tasks even in a simple 2D design space. And even though our current sensing approach is simplistic, adding extra complexity to our treatment of sensing might retract from the main goal of the co-design algorithms and make them struggle unnecessarily. As co-design methods improve in the future, we may be able to introduce more realistic simulation techniques like complicated sensing and real-world obstacles such as noise, eventually bridging the sim-to-real gap.

-------------------------------------------------------------------------------------------------

Overall, our carefully designed benchmarking tool fills an important missing piece in research in soft robotics and robotic evolution algorithms. We believe NeurIPS would provide the perfect launching pad to expose our tool to the types of researchers equipped and excited to utilize such a benchmark.

We hope in this response we have clarified the goals of our original paper and adequately addressed your concerns. Your comments have helped us strengthen the work. If you have any other questions, we would be open to further discussion. 

Sincerely,

The authors of Evolution Gym

[1] Zhang, Chao, et al. “Modular Soft Robotics: Modular Units, Connection Mechanisms, and Applications.” Advanced Intelligent Systems (2020).

[2] Langford, Will. “Discrete Robotic Construction.” PhD Thesis (2019).

[3] Romashin et al. “Decentralized Control for 3D M-Blocks for Path Following, Line Formation, and Light Gradient Aggregation.” IEEE IROS (2021).

[4] Cramer, Nick et al. “Design and Testing of FERVOR: FlexiblE and Reconfigurable Voxel-based Robot,” IEEE IROS (2017).

[5] Wehner, Michael. “An integrated design and fabrication strategy for entirely soft, autonomous robots.” Nature (2016).


The paper presents a benchmark for soft robotics along with several baseline implementations. The benchmark creation is carefully designed and it is clear that it will be a useful resource for the Evolution robotics, RL, and co-adaptation community.  This paper solves an important problem -- a benchmark + standard baseline implementation for soft robot evolution. The benchmark is carefully constructed and follows a line of past research. The ultimate test is how well and often do people use it as a benchmark. I do not have any serious objections to this paper. A few minor questions and feedback are as follows.

1. Lines 30-31. I am not sure if this is entirely accurate. Some recent work such as Hejna et al. 2021 shows co-adaptation for arbitrary tasks.
2. Time analysis. How slow/fast is the simulator compared to mujoco or other standard benchmark?
3. 2d vs 3d. Please comment on extending such environments for 3D environments.

 N/A Early stage research.

In the paper, "Evolution Gym: A Large-Scale Benchmark for Evolving Soft Robots", the authors introduce a new framework for the benchmark for the evolution of the design of soft robots. This framework, called "Evolution Gym, contains 30 environments (only 10 are described in the main paper), which cover different types of tasks (locomotion, manipulation, and a mixture of both) and different level of difficulties. In addition to the introduction of the framework, the paper also presents experimental results in which the authors have implemented three baseline algorithms (Genetic Algorithm, Bayesian Optimisation, and CPPN-NEAT) for the optimisation of the robot's morphology and PPO for the optimisation of the controllers.  The paper is well written, clearly structured and easy to follow. The illustrations are also very nice and enable the readers to easily understand the objective of the different tasks. I believe that the central idea of the paper of introducing a novel common benchmark for the evolution of soft robots is a great idea. I hope this will have the same impact as the OpenAI Gym environment had on the reinforcement learning community. Overall, I enjoyed reading the paper.

However, I have two main concerns with the paper. I structured these comments in points and subpoints to make future discussions easier:
1) I expected the paper to be focused on the proposed framework/benchmark implementation: 
1.1) In particular, there are some claims like "user-friendly Python interfaces", "expressive multi-material voxel-based structure design space", "flexibility of the controller parametrisation" that are not much supported by the paper. What makes the interface more or less user friendly than other tools? What are the limits, or advantages on the expressivity of the morphological encoding used in this benchmark compared to the literature? Is it using the state-of-the-art or even innovating on top of it? Etc.
1.2) An important aspect of such tools is the documentation, but nothing is said about this in the paper. 
1.3) After reading the paper, there are multiple questions that remain unanswered to me: 
What are the advantages of using this framework compared to other existing ones? Is it offering any performance improvement over alternatives? 

2) The experimental comparison is not adding much to the paper when we consider this is taking more than a third of the paper. The considered baselines are not novel, but instead, well-established algorithms (as clearly stated by the authors). This is an excellent point to provide existing baselines with such benchmark tools (similar to Stable baselines in Deep RL). 
2.1) However, I believe it would have been interesting to link these algorithms with existing papers attempting to apply these baselines in similar domains and demonstrating that the implemented baselines can act as reproductions of these previous results. This would provide tools to researchers to try to build on top of these algorithms. 
2.2) The main outcome of the experimental section is in my opinion the relative difficulty order of the different tasks. It is particularly interesting to see that some tasks remain unsolved, which represents a nice challenge for future work. However, it seems that we are observing the expected results based on the literature (i.e., BO does not scale well, GA is slow but can lead to very good results and CPPN-NEAT is very domain-dependent). 

Overall, the paper proposes an excellent tool for researchers. Yet, I believe the paper can be made significantly stronger by focusing more on the framework/benchmark and its advantages (which needs to be demonstrated) than on the experimental evaluation of well-established algorithms.  The authors discuss the limitations of their approach and I cannot see any direct societal impact of their work.

Meta-review: The paper proposes a simulation-based benchmark for learning and evaluating algorithms that jointly evolve the design and control of soft robot bodies. The benchmark includes a suite of environments involving locomotion and manipulation tasks. The benchmark includes implementations of several existing design-control optimization methods and the paper presents experimental results for these baselines on a subset of the tasks.

Jointly optimizing a robot's physical design together with its control policy is a challenging problem that has recently gotten the attention of many researchers in the machine learning community. These challenges are exacerbated in the case of soft-bodied robots due to the significantly richer design space and the complexity of the physics. Owing to this and the renewed attention paid to joint optimization of design+control, the availability of a benchmark suite for soft robots would provide a valuable contribution. The reviewers agree that the paper is well motivated and well written. However, there are concerns about some of the assumptions and claims made in the paper, notably the sufficiency of a voxel-based representation of the design space, the simplistic physics, and a lack of a discussion of the sim-to-real gap, which seems to be exacerbated by the simplicity of the physics model and the nature of the design space. Thus, it is unclear which communities this benchmark is targeting. One can certainly see it being used as a simulation benchmark suite for the (robot) learning community, but the extent to which it will benefit soft roboticists appears limited.

Decision: accept