Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes Evolution Gym, a simulation environment and benchmark which may be used for evaluating design and control optimization techniques for soft robots built in a voxelized design space. In addition to the benchmark, the paper compares several existing design and control optimization techniques for a common set of tasks.
main_review: The paper posits that co-design algorithms are understudied due to the complex, bi-level nature of the underlying problem and the lack of a benchmarking platform. It seems that a benchmarking platform which could be used for co-design of robots would be a useful tool to roboticists. In my opinion, such a platform could be used to gain understanding of some key concepts underlying soft robots (e.g. the role of morphological intelligence) while providing clear baselines of optimization techniques and behavior which will establish a state-of-the-art. This contribution has the potential to be important to soft robotics.

However, some aspects of the provided benchmark and comparison are underdeveloped. In general, a stronger case could have been made for the particular choice of physics, design space, and the presence of feedback in the proposed benchmark. Further, some clarifications are needed about the optimization comparison.

1. The approach given has a key sim-to-real limitation that is not discussed. Details are given in the following section.

2. Why are voxel-based design spaces the best choice? It is claimed that voxel-based soft robots offer a "general and universal representation" of various "categories" of robot designs, yet it is not clear which categories the learned robots fit into. Indeed, it is not even clear at present how to physically construct voxel-based soft robots under this general representation (e.g. Kriegman et al, 2020, Scalable sim-to-real transfer of soft robot designs). Authors might consider commenting on learned systems, noting whether actuators presently exist that allow these robots to be constructed. Is it better to have a generalized representation such as this one, or many possible constrained design spaces that are more directly inspired from existing soft robot designs? Could robots learned in Evolution Gym guide soft actuator development? Detailed discussion of this point could help readers.

3. Why does a benchmark need to include feedback control? The paper notes that prior co-design work in soft robotics mainly has mainly explored open-loop control, while the proposed framework allows detailed high-dimensional observations to be used as feedback. This engages directly with ideas of morphological computation (e.g. Tolley and Rus, 2015) and its role in soft robotics; the simulator could allow experiments about which balance between morphological computation and feedback is best. Yet, here, it is not clear how much feedback the learned controllers are using and whether the inclusion of it was needed. Comments about this issue, and how the proposed benchmark may help with its study, would better show the impact of the work.

4. It seems somewhat surprising that the plain GA outperforms the other methods in several tasks, given that GAs have not been shown to converge to global minima. Comments on this success might help readers.

5. Is it possible to use gradient-based methods for design optimization with this benchmark? Would anyone need to do so in the given design space? Please clarify.

6. Why is a simplified, 2D spring-based physics representation best? It is clear that such an approach is faster computationally than other mechanics methods (e.g. Finite Element Analysis or particle-based methods), but accuracy also suffers if the approach is simplified. Why not take a more computationally heavy (but accurate) approach and wait for Moore's Law? Please discuss. Further, does this physical simulation differ from the one used by Medvet et al 2020 (citation 21)? Please clarify.
limitations_and_societal_impact: One key limitation of a this simplified benchmark environment is that its gap with reality is not yet characterized. The paper states that "we observed how intelligent robots could be evolved autonomously from scratch yet still be capable of accomplishing some surprisingly complex tasks." While this did happen in the simulation environment, it is not clear that real devices built with the best-performing designs and controllers would truly perform best in the real world. As noted above, it is unclear whether the result robots are easy to construct, and it is further unclear whether sensing technologies are possible for the type of observations described in Section 3.3. While a detailed experimental validation of all proposed robots is clearly out of scope, some comments on the reality gap and the transition from 2D to 3D are 
necessary given the embodied nature of the problem.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper presents a benchmark for soft robotics along with several baseline implementations. The benchmark creation is carefully designed and it is clear that it will be a useful resource for the Evolution robotics, RL, and co-adaptation community.
main_review: This paper solves an important problem -- a benchmark + standard baseline implementation for soft robot evolution. The benchmark is carefully constructed and follows a line of past research. The ultimate test is how well and often do people use it as a benchmark. I do not have any serious objections to this paper. A few minor questions and feedback are as follows.

1. Lines 30-31. I am not sure if this is entirely accurate. Some recent work such as Hejna et al. 2021 shows co-adaptation for arbitrary tasks.
2. Time analysis. How slow/fast is the simulator compared to mujoco or other standard benchmark?
3. 2d vs 3d. Please comment on extending such environments for 3D environments.


limitations_and_societal_impact: N/A Early stage research.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: In the paper, "Evolution Gym: A Large-Scale Benchmark for Evolving Soft Robots", the authors introduce a new framework for the benchmark for the evolution of the design of soft robots. This framework, called "Evolution Gym, contains 30 environments (only 10 are described in the main paper), which cover different types of tasks (locomotion, manipulation, and a mixture of both) and different level of difficulties. In addition to the introduction of the framework, the paper also presents experimental results in which the authors have implemented three baseline algorithms (Genetic Algorithm, Bayesian Optimisation, and CPPN-NEAT) for the optimisation of the robot's morphology and PPO for the optimisation of the controllers.
main_review: The paper is well written, clearly structured and easy to follow. The illustrations are also very nice and enable the readers to easily understand the objective of the different tasks. I believe that the central idea of the paper of introducing a novel common benchmark for the evolution of soft robots is a great idea. I hope this will have the same impact as the OpenAI Gym environment had on the reinforcement learning community. Overall, I enjoyed reading the paper.

However, I have two main concerns with the paper. I structured these comments in points and subpoints to make future discussions easier:
1) I expected the paper to be focused on the proposed framework/benchmark implementation: 
1.1) In particular, there are some claims like "user-friendly Python interfaces", "expressive multi-material voxel-based structure design space", "flexibility of the controller parametrisation" that are not much supported by the paper. What makes the interface more or less user friendly than other tools? What are the limits, or advantages on the expressivity of the morphological encoding used in this benchmark compared to the literature? Is it using the state-of-the-art or even innovating on top of it? Etc.
1.2) An important aspect of such tools is the documentation, but nothing is said about this in the paper. 
1.3) After reading the paper, there are multiple questions that remain unanswered to me: 
What are the advantages of using this framework compared to other existing ones? Is it offering any performance improvement over alternatives? 

2) The experimental comparison is not adding much to the paper when we consider this is taking more than a third of the paper. The considered baselines are not novel, but instead, well-established algorithms (as clearly stated by the authors). This is an excellent point to provide existing baselines with such benchmark tools (similar to Stable baselines in Deep RL). 
2.1) However, I believe it would have been interesting to link these algorithms with existing papers attempting to apply these baselines in similar domains and demonstrating that the implemented baselines can act as reproductions of these previous results. This would provide tools to researchers to try to build on top of these algorithms. 
2.2) The main outcome of the experimental section is in my opinion the relative difficulty order of the different tasks. It is particularly interesting to see that some tasks remain unsolved, which represents a nice challenge for future work. However, it seems that we are observing the expected results based on the literature (i.e., BO does not scale well, GA is slow but can lead to very good results and CPPN-NEAT is very domain-dependent). 

Overall, the paper proposes an excellent tool for researchers. Yet, I believe the paper can be made significantly stronger by focusing more on the framework/benchmark and its advantages (which needs to be demonstrated) than on the experimental evaluation of well-established algorithms. 
limitations_and_societal_impact: The authors discuss the limitations of their approach and I cannot see any direct societal impact of their work.
review_title: null

Meta-review: The paper proposes a simulation-based benchmark for learning and evaluating algorithms that jointly evolve the design and control of soft robot bodies. The benchmark includes a suite of environments involving locomotion and manipulation tasks. The benchmark includes implementations of several existing design-control optimization methods and the paper presents experimental results for these baselines on a subset of the tasks.

Jointly optimizing a robot's physical design together with its control policy is a challenging problem that has recently gotten the attention of many researchers in the machine learning community. These challenges are exacerbated in the case of soft-bodied robots due to the significantly richer design space and the complexity of the physics. Owing to this and the renewed attention paid to joint optimization of design+control, the availability of a benchmark suite for soft robots would provide a valuable contribution. The reviewers agree that the paper is well motivated and well written. However, there are concerns about some of the assumptions and claims made in the paper, notably the sufficiency of a voxel-based representation of the design space, the simplistic physics, and a lack of a discussion of the sim-to-real gap, which seems to be exacerbated by the simplicity of the physics model and the nature of the design space. Thus, it is unclear which communities this benchmark is targeting. One can certainly see it being used as a simulation benchmark suite for the (robot) learning community, but the extent to which it will benefit soft roboticists appears limited.
Decision: Accept (Poster)